Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 07:58:26.661949 398 test_util.cc:268] Using random seed: -637901276
I20240501 07:58:26.669780 398 ts_itest-base.cc:112] Starting cluster with:
I20240501 07:58:26.669852 398 ts_itest-base.cc:113] --------------
I20240501 07:58:26.669867 398 ts_itest-base.cc:114] 8 tablet servers
I20240501 07:58:26.669884 398 ts_itest-base.cc:115] 3 replicas per TS
I20240501 07:58:26.669898 398 ts_itest-base.cc:116] --------------
2024-05-01T07:58:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T07:58:26Z Disabled control of system clock
I20240501 07:58:26.679905 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:35609
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:35609
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 07:58:26.793906 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:26.794185 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:26.794232 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:26.797607 410 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:26.797711 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 07:58:26.797736 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:26.797755 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 07:58:26.797770 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 07:58:26.801968 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:35609
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:35609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:26.803028 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:26.804484 410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:26.813544 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:26.814043 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:26.814582 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:26.815349 410 server_base.cc:1034] running on GCE node
I20240501 07:58:26.815992 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:26.817117 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:26.818312 410 hybrid_clock.cc:648] HybridClock initialized: now 1714550306818250 us; error 82 us; skew 500 ppm
I20240501 07:58:26.824414 410 webserver.cc:460] Webserver started at http://127.0.99.190:37643/ using document root <none> and password file <none>
I20240501 07:58:26.825129 410 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:26.825210 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:26.825408 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:26.827291 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f74f7e461feb4fd681f0bf8f33c2c623"
format_stamp: "Formatted at 2024-05-01 07:58:26 on dist-test-slave-2srn"
I20240501 07:58:26.827823 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f74f7e461feb4fd681f0bf8f33c2c623"
format_stamp: "Formatted at 2024-05-01 07:58:26 on dist-test-slave-2srn"
I20240501 07:58:26.831451 410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:58:26.834090 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:26.835300 410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:26.835433 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f74f7e461feb4fd681f0bf8f33c2c623"
format_stamp: "Formatted at 2024-05-01 07:58:26 on dist-test-slave-2srn"
I20240501 07:58:26.835532 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:26.857167 410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:26.857879 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:26.858060 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:26.865371 410 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:35609
I20240501 07:58:26.865382 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:35609 every 8 connection(s)
I20240501 07:58:26.866544 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 07:58:26.869380 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:26.874859 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623: Bootstrap starting.
I20240501 07:58:26.876830 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 410
I20240501 07:58:26.876991 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 07:58:26.877604 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:26.878526 477 log.cc:824] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:26.880648 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623: No bootstrap required, opened a new log
I20240501 07:58:26.883952 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74f7e461feb4fd681f0bf8f33c2c623" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35609 } }
I20240501 07:58:26.884207 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:26.884310 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74f7e461feb4fd681f0bf8f33c2c623, State: Initialized, Role: FOLLOWER
I20240501 07:58:26.884824 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74f7e461feb4fd681f0bf8f33c2c623" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35609 } }
I20240501 07:58:26.884986 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:26.885092 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:26.885205 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:26.899402 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74f7e461feb4fd681f0bf8f33c2c623" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35609 } }
I20240501 07:58:26.899966 477 leader_election.cc:302] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f74f7e461feb4fd681f0bf8f33c2c623; no voters:
I20240501 07:58:26.900327 477 leader_election.cc:288] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:26.900566 482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:26.900768 482 raft_consensus.cc:686] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623 [term 1 LEADER]: Becoming Leader. State: Replica: f74f7e461feb4fd681f0bf8f33c2c623, State: Running, Role: LEADER
I20240501 07:58:26.901170 482 consensus_queue.cc:237] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74f7e461feb4fd681f0bf8f33c2c623" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35609 } }
I20240501 07:58:26.901728 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:58:26.903272 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f74f7e461feb4fd681f0bf8f33c2c623" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74f7e461feb4fd681f0bf8f33c2c623" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35609 } } }
I20240501 07:58:26.903409 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623 [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:26.904345 486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:58:26.904595 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f74f7e461feb4fd681f0bf8f33c2c623. Latest consensus state: current_term: 1 leader_uuid: "f74f7e461feb4fd681f0bf8f33c2c623" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74f7e461feb4fd681f0bf8f33c2c623" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35609 } } }
I20240501 07:58:26.904799 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623 [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:26.907625 486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:58:26.914016 486 catalog_manager.cc:1347] Generated new cluster ID: d61a8d1e8ccd4fc6934a34701ac528c5
I20240501 07:58:26.914117 486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:58:26.932191 486 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:58:26.933216 486 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:58:26.939468 486 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f74f7e461feb4fd681f0bf8f33c2c623: Generated new TSK 0
I20240501 07:58:26.940440 486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:58:26.953799 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35609
--builtin_ntp_servers=127.0.99.148:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 07:58:27.075171 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:27.075433 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:27.075503 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:27.079044 501 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:27.079144 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:27.079247 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240501 07:58:27.083277 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35609
--never_fsync=true
--heap_profile_path=/tmp/kudu.501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:27.084295 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:27.085534 501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:27.094614 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.095029 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.095499 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:27.096087 501 server_base.cc:1034] running on GCE node
I20240501 07:58:27.096590 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:27.097218 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:27.098439 501 hybrid_clock.cc:648] HybridClock initialized: now 1714550307098386 us; error 53 us; skew 500 ppm
I20240501 07:58:27.100661 501 webserver.cc:460] Webserver started at http://127.0.99.129:42113/ using document root <none> and password file <none>
I20240501 07:58:27.101673 501 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:27.101745 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:27.101933 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:27.103811 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-2srn"
I20240501 07:58:27.104337 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-2srn"
I20240501 07:58:27.108086 501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 07:58:27.110633 515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.111807 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:27.111949 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-2srn"
I20240501 07:58:27.112052 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:27.147233 501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:27.147914 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:27.148084 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:27.148681 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:27.149622 501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:27.149678 501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.149750 501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:27.149776 501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.158746 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:35849
I20240501 07:58:27.158777 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:35849 every 8 connection(s)
I20240501 07:58:27.159767 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 07:58:27.166528 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 501
I20240501 07:58:27.166657 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 07:58:27.169144 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35609
--builtin_ntp_servers=127.0.99.148:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 07:58:27.169719 629 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35609
I20240501 07:58:27.169983 629 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:27.170678 629 heartbeater.cc:511] Master 127.0.99.190:35609 requested a full tablet report, sending...
I20240501 07:58:27.172335 440 ts_manager.cc:194] Registered new tserver with Master: 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849)
I20240501 07:58:27.173537 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:58265
W20240501 07:58:27.285586 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:27.285851 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:27.285910 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:27.289527 632 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:27.289625 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:27.289709 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240501 07:58:27.293717 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35609
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:27.294837 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:27.296046 632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:27.307073 640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.307587 637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.308140 638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:27.308712 632 server_base.cc:1034] running on GCE node
I20240501 07:58:27.309460 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:27.310124 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:27.311305 632 hybrid_clock.cc:648] HybridClock initialized: now 1714550307311277 us; error 45 us; skew 500 ppm
I20240501 07:58:27.313680 632 webserver.cc:460] Webserver started at http://127.0.99.130:37151/ using document root <none> and password file <none>
I20240501 07:58:27.314311 632 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:27.314386 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:27.314607 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:27.316576 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e2c804beb8ef40ecb16c952f8875ca02"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-2srn"
I20240501 07:58:27.317135 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e2c804beb8ef40ecb16c952f8875ca02"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-2srn"
I20240501 07:58:27.321110 632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 07:58:27.323845 646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.325158 632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:27.325289 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e2c804beb8ef40ecb16c952f8875ca02"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-2srn"
I20240501 07:58:27.325485 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:27.338863 632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:27.339535 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:27.339720 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:27.340314 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:27.341234 632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:27.341293 632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.341347 632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:27.341377 632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.351902 632 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:43577
I20240501 07:58:27.351974 759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:43577 every 8 connection(s)
I20240501 07:58:27.352972 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 07:58:27.356169 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 632
I20240501 07:58:27.356299 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 07:58:27.360834 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35609
--builtin_ntp_servers=127.0.99.148:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 07:58:27.368786 760 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35609
I20240501 07:58:27.369202 760 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:27.370226 760 heartbeater.cc:511] Master 127.0.99.190:35609 requested a full tablet report, sending...
I20240501 07:58:27.371783 440 ts_manager.cc:194] Registered new tserver with Master: e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577)
I20240501 07:58:27.372613 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:38599
W20240501 07:58:27.491273 763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:27.491562 763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:27.491650 763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:27.495069 763 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:27.495177 763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:27.495283 763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240501 07:58:27.499449 763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35609
--never_fsync=true
--heap_profile_path=/tmp/kudu.763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:27.500469 763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:27.501717 763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:27.511379 768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.511620 771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.512126 769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:27.512768 763 server_base.cc:1034] running on GCE node
I20240501 07:58:27.513204 763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:27.513806 763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:27.514981 763 hybrid_clock.cc:648] HybridClock initialized: now 1714550307514953 us; error 45 us; skew 500 ppm
I20240501 07:58:27.517387 763 webserver.cc:460] Webserver started at http://127.0.99.131:36971/ using document root <none> and password file <none>
I20240501 07:58:27.518020 763 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:27.518087 763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:27.518290 763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:27.520118 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "10aab83192f447829c9b8254e36d94b4"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-2srn"
I20240501 07:58:27.520651 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "10aab83192f447829c9b8254e36d94b4"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-2srn"
I20240501 07:58:27.524564 763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240501 07:58:27.527156 777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.528340 763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:27.528487 763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "10aab83192f447829c9b8254e36d94b4"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-2srn"
I20240501 07:58:27.528597 763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:27.546759 763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:27.547389 763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:27.547561 763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:27.548149 763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:27.549096 763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:27.549147 763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.549189 763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:27.549216 763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.558461 763 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:40475
I20240501 07:58:27.558509 890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:40475 every 8 connection(s)
I20240501 07:58:27.559463 763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 07:58:27.568079 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 763
I20240501 07:58:27.568195 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 07:58:27.569595 891 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35609
I20240501 07:58:27.569864 891 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:27.570490 891 heartbeater.cc:511] Master 127.0.99.190:35609 requested a full tablet report, sending...
I20240501 07:58:27.570618 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35609
--builtin_ntp_servers=127.0.99.148:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 07:58:27.571751 437 ts_manager.cc:194] Registered new tserver with Master: 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475)
I20240501 07:58:27.572597 437 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:42865
W20240501 07:58:27.692157 894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:27.692409 894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:27.692458 894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:27.695931 894 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:27.696029 894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:27.696116 894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:58:27.700145 894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35609
--never_fsync=true
--heap_profile_path=/tmp/kudu.894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:27.701131 894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:27.702361 894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:27.711391 899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.711872 900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.712285 902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:27.712940 894 server_base.cc:1034] running on GCE node
I20240501 07:58:27.713379 894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:27.713963 894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:27.715144 894 hybrid_clock.cc:648] HybridClock initialized: now 1714550307715129 us; error 38 us; skew 500 ppm
I20240501 07:58:27.717321 894 webserver.cc:460] Webserver started at http://127.0.99.132:40147/ using document root <none> and password file <none>
I20240501 07:58:27.717896 894 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:27.717955 894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:27.718130 894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:27.719882 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e2edc87d834f4e0bb662d501af69f2b5"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-2srn"
I20240501 07:58:27.720371 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e2edc87d834f4e0bb662d501af69f2b5"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-2srn"
I20240501 07:58:27.723994 894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 07:58:27.726516 908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.727646 894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:27.727775 894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e2edc87d834f4e0bb662d501af69f2b5"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-2srn"
I20240501 07:58:27.727880 894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:27.743305 894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:27.743991 894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:27.744179 894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:27.744768 894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:27.745733 894 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:27.745787 894 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.745848 894 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:27.745878 894 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.756176 894 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:36973
I20240501 07:58:27.756219 1021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:36973 every 8 connection(s)
I20240501 07:58:27.757416 894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:58:27.757885 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 894
I20240501 07:58:27.757989 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 07:58:27.760462 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35609
--builtin_ntp_servers=127.0.99.148:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 07:58:27.769990 1022 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35609
I20240501 07:58:27.770314 1022 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:27.771234 1022 heartbeater.cc:511] Master 127.0.99.190:35609 requested a full tablet report, sending...
I20240501 07:58:27.772630 437 ts_manager.cc:194] Registered new tserver with Master: e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973)
I20240501 07:58:27.773241 437 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:45267
W20240501 07:58:27.882977 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:27.883332 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:27.883415 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:27.888072 1025 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:27.888175 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:27.888298 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:58:27.893431 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35609
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:27.894721 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:27.896306 1025 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:27.907160 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.907510 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.908560 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:27.908597 1025 server_base.cc:1034] running on GCE node
I20240501 07:58:27.909080 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:27.909806 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:27.910944 1025 hybrid_clock.cc:648] HybridClock initialized: now 1714550307910874 us; error 90 us; skew 500 ppm
I20240501 07:58:27.913409 1025 webserver.cc:460] Webserver started at http://127.0.99.133:33613/ using document root <none> and password file <none>
I20240501 07:58:27.914017 1025 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:27.914081 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:27.914290 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:27.916236 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b331ee755f7d4df48eb9f4460a0340b6"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-2srn"
I20240501 07:58:27.916808 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b331ee755f7d4df48eb9f4460a0340b6"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-2srn"
I20240501 07:58:27.920765 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 07:58:27.923373 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.924609 1025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:27.924728 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b331ee755f7d4df48eb9f4460a0340b6"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-2srn"
I20240501 07:58:27.924887 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:27.948253 1025 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:27.948917 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:27.949095 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:27.949687 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:27.950692 1025 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:27.950753 1025 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.950803 1025 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:27.950824 1025 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.961367 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:39953
I20240501 07:58:27.961520 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:39953 every 8 connection(s)
I20240501 07:58:27.962540 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:58:27.968238 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 1025
I20240501 07:58:27.968336 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 07:58:27.970775 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35609
--builtin_ntp_servers=127.0.99.148:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 07:58:27.972627 1153 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35609
I20240501 07:58:27.972891 1153 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:27.973491 1153 heartbeater.cc:511] Master 127.0.99.190:35609 requested a full tablet report, sending...
I20240501 07:58:27.974678 437 ts_manager.cc:194] Registered new tserver with Master: b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133:39953)
I20240501 07:58:27.975322 437 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:43367
W20240501 07:58:28.089084 1156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:28.089365 1156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:28.089418 1156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:28.092638 1156 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:28.092731 1156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:28.092836 1156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 07:58:28.096691 1156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35609
--never_fsync=true
--heap_profile_path=/tmp/kudu.1156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:28.097748 1156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:28.098906 1156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:28.107764 1164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:28.108247 1162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:28.108681 1161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:28.109261 1156 server_base.cc:1034] running on GCE node
I20240501 07:58:28.109690 1156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:28.110311 1156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:28.111460 1156 hybrid_clock.cc:648] HybridClock initialized: now 1714550308111424 us; error 56 us; skew 500 ppm
I20240501 07:58:28.113488 1156 webserver.cc:460] Webserver started at http://127.0.99.134:45131/ using document root <none> and password file <none>
I20240501 07:58:28.114086 1156 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:28.114145 1156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:28.114321 1156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:28.116142 1156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "36272be7c71b413a94d36b4257d34425"
format_stamp: "Formatted at 2024-05-01 07:58:28 on dist-test-slave-2srn"
I20240501 07:58:28.116663 1156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "36272be7c71b413a94d36b4257d34425"
format_stamp: "Formatted at 2024-05-01 07:58:28 on dist-test-slave-2srn"
I20240501 07:58:28.120277 1156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 07:58:28.122695 1170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:28.123871 1156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:28.124006 1156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "36272be7c71b413a94d36b4257d34425"
format_stamp: "Formatted at 2024-05-01 07:58:28 on dist-test-slave-2srn"
I20240501 07:58:28.124114 1156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:28.137986 1156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:28.138777 1156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:28.138953 1156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:28.139560 1156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:28.140650 1156 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:28.140698 1156 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:28.140755 1156 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:28.140781 1156 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:28.150296 1156 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:39945
I20240501 07:58:28.150323 1283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:39945 every 8 connection(s)
I20240501 07:58:28.151312 1156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 07:58:28.158789 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 1156
I20240501 07:58:28.158900 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 07:58:28.160387 1284 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35609
I20240501 07:58:28.160640 1284 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:28.161126 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35609
--builtin_ntp_servers=127.0.99.148:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 07:58:28.161221 1284 heartbeater.cc:511] Master 127.0.99.190:35609 requested a full tablet report, sending...
I20240501 07:58:28.163501 437 ts_manager.cc:194] Registered new tserver with Master: 36272be7c71b413a94d36b4257d34425 (127.0.99.134:39945)
I20240501 07:58:28.164311 437 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:46543
I20240501 07:58:28.176566 629 heartbeater.cc:503] Master 127.0.99.190:35609 was elected leader, sending a full tablet report...
W20240501 07:58:28.280097 1287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:28.280385 1287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:28.280439 1287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:28.283814 1287 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:28.283915 1287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:28.283999 1287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 07:58:28.288107 1287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35609
--never_fsync=true
--heap_profile_path=/tmp/kudu.1287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:28.289033 1287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:28.290239 1287 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:28.299134 1293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:28.299682 1295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:28.300097 1292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:28.300741 1287 server_base.cc:1034] running on GCE node
I20240501 07:58:28.301205 1287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:28.301887 1287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:28.303071 1287 hybrid_clock.cc:648] HybridClock initialized: now 1714550308303036 us; error 49 us; skew 500 ppm
I20240501 07:58:28.305608 1287 webserver.cc:460] Webserver started at http://127.0.99.135:40893/ using document root <none> and password file <none>
I20240501 07:58:28.306227 1287 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:28.306296 1287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:28.306528 1287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:28.308367 1287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "c1b68243a7144af5835b051aa9916d13"
format_stamp: "Formatted at 2024-05-01 07:58:28 on dist-test-slave-2srn"
I20240501 07:58:28.308919 1287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "c1b68243a7144af5835b051aa9916d13"
format_stamp: "Formatted at 2024-05-01 07:58:28 on dist-test-slave-2srn"
I20240501 07:58:28.312999 1287 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 07:58:28.315774 1301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:28.317113 1287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:28.317232 1287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "c1b68243a7144af5835b051aa9916d13"
format_stamp: "Formatted at 2024-05-01 07:58:28 on dist-test-slave-2srn"
I20240501 07:58:28.317346 1287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:28.336244 1287 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:28.336899 1287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:28.337059 1287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:28.337610 1287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:28.338572 1287 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:28.338621 1287 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:28.338663 1287 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:28.338689 1287 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:28.349260 1287 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:44501
I20240501 07:58:28.349283 1414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:44501 every 8 connection(s)
I20240501 07:58:28.350276 1287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 07:58:28.358870 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 1287
I20240501 07:58:28.358970 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 07:58:28.359830 1415 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35609
I20240501 07:58:28.360070 1415 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:28.360692 1415 heartbeater.cc:511] Master 127.0.99.190:35609 requested a full tablet report, sending...
I20240501 07:58:28.360980 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35609
--builtin_ntp_servers=127.0.99.148:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 07:58:28.361840 437 ts_manager.cc:194] Registered new tserver with Master: c1b68243a7144af5835b051aa9916d13 (127.0.99.135:44501)
I20240501 07:58:28.362646 437 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:57103
I20240501 07:58:28.375551 760 heartbeater.cc:503] Master 127.0.99.190:35609 was elected leader, sending a full tablet report...
W20240501 07:58:28.479524 1418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:28.479784 1418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:28.479833 1418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:28.483108 1418 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:28.483207 1418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:28.483284 1418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240501 07:58:28.487520 1418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35609
--never_fsync=true
--heap_profile_path=/tmp/kudu.1418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:28.488559 1418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:28.489761 1418 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:28.500270 1424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:28.500727 1426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:28.501139 1423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:28.501757 1418 server_base.cc:1034] running on GCE node
I20240501 07:58:28.502272 1418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:28.502987 1418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:28.504171 1418 hybrid_clock.cc:648] HybridClock initialized: now 1714550308504148 us; error 39 us; skew 500 ppm
I20240501 07:58:28.506584 1418 webserver.cc:460] Webserver started at http://127.0.99.136:34637/ using document root <none> and password file <none>
I20240501 07:58:28.507202 1418 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:28.507268 1418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:28.507454 1418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:28.512302 1418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "6645111fc9e446e8877750a3c4779750"
format_stamp: "Formatted at 2024-05-01 07:58:28 on dist-test-slave-2srn"
I20240501 07:58:28.512881 1418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "6645111fc9e446e8877750a3c4779750"
format_stamp: "Formatted at 2024-05-01 07:58:28 on dist-test-slave-2srn"
I20240501 07:58:28.516795 1418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20240501 07:58:28.519496 1432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:28.520700 1418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240501 07:58:28.520856 1418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "6645111fc9e446e8877750a3c4779750"
format_stamp: "Formatted at 2024-05-01 07:58:28 on dist-test-slave-2srn"
I20240501 07:58:28.520964 1418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:28.533277 1418 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:28.533974 1418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:28.534139 1418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:28.534766 1418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:28.535714 1418 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:28.535765 1418 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:28.535821 1418 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:28.535851 1418 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:28.546283 1418 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:44643
I20240501 07:58:28.546449 1545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:44643 every 8 connection(s)
I20240501 07:58:28.547330 1418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 07:58:28.548260 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 1418
I20240501 07:58:28.548343 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 07:58:28.557610 1546 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35609
I20240501 07:58:28.557891 1546 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:28.558630 1546 heartbeater.cc:511] Master 127.0.99.190:35609 requested a full tablet report, sending...
I20240501 07:58:28.559809 437 ts_manager.cc:194] Registered new tserver with Master: 6645111fc9e446e8877750a3c4779750 (127.0.99.136:44643)
I20240501 07:58:28.560395 437 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:35795
I20240501 07:58:28.563771 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 07:58:28.575796 891 heartbeater.cc:503] Master 127.0.99.190:35609 was elected leader, sending a full tablet report...
I20240501 07:58:28.579007 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 07:58:28.579128 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 1025
I20240501 07:58:28.584429 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 1156
I20240501 07:58:28.592494 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 1287
I20240501 07:58:28.597932 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 1418
I20240501 07:58:28.776093 1022 heartbeater.cc:503] Master 127.0.99.190:35609 was elected leader, sending a full tablet report...
I20240501 07:58:32.355443 437 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35336:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:32.378346 563 tablet_service.cc:1467] Processing CreateTablet for tablet 2c969b6e50fd4fd4919c181ae51a47ae (DEFAULT_TABLE table=rebalance_test_table_0 [id=cef0e65e05d34b97905a5928033946ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.378346 824 tablet_service.cc:1467] Processing CreateTablet for tablet 5d48c8225721471f9eea917078d5ef2d (DEFAULT_TABLE table=rebalance_test_table_0 [id=cef0e65e05d34b97905a5928033946ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.378345 694 tablet_service.cc:1467] Processing CreateTablet for tablet c6f89045c20f4ccbb525febd8f5ce56e (DEFAULT_TABLE table=rebalance_test_table_0 [id=cef0e65e05d34b97905a5928033946ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.379168 562 tablet_service.cc:1467] Processing CreateTablet for tablet c6f89045c20f4ccbb525febd8f5ce56e (DEFAULT_TABLE table=rebalance_test_table_0 [id=cef0e65e05d34b97905a5928033946ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.379492 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6f89045c20f4ccbb525febd8f5ce56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.379491 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c969b6e50fd4fd4919c181ae51a47ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.379491 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d48c8225721471f9eea917078d5ef2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.379642 561 tablet_service.cc:1467] Processing CreateTablet for tablet 5d48c8225721471f9eea917078d5ef2d (DEFAULT_TABLE table=rebalance_test_table_0 [id=cef0e65e05d34b97905a5928033946ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.379848 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d48c8225721471f9eea917078d5ef2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.378345 825 tablet_service.cc:1467] Processing CreateTablet for tablet c6f89045c20f4ccbb525febd8f5ce56e (DEFAULT_TABLE table=rebalance_test_table_0 [id=cef0e65e05d34b97905a5928033946ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.380796 825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6f89045c20f4ccbb525febd8f5ce56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.381160 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6f89045c20f4ccbb525febd8f5ce56e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.382673 693 tablet_service.cc:1467] Processing CreateTablet for tablet 2c969b6e50fd4fd4919c181ae51a47ae (DEFAULT_TABLE table=rebalance_test_table_0 [id=cef0e65e05d34b97905a5928033946ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.382962 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c969b6e50fd4fd4919c181ae51a47ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.387053 1566 tablet_bootstrap.cc:492] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:32.388249 1568 tablet_bootstrap.cc:492] T 2c969b6e50fd4fd4919c181ae51a47ae P e2c804beb8ef40ecb16c952f8875ca02: Bootstrap starting.
I20240501 07:58:32.389667 1566 tablet_bootstrap.cc:654] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.390558 1568 tablet_bootstrap.cc:654] T 2c969b6e50fd4fd4919c181ae51a47ae P e2c804beb8ef40ecb16c952f8875ca02: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.390625 1566 log.cc:824] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:32.391469 1567 tablet_bootstrap.cc:492] T 5d48c8225721471f9eea917078d5ef2d P 10aab83192f447829c9b8254e36d94b4: Bootstrap starting.
I20240501 07:58:32.391736 1568 log.cc:824] T 2c969b6e50fd4fd4919c181ae51a47ae P e2c804beb8ef40ecb16c952f8875ca02: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:32.393386 1566 tablet_bootstrap.cc:492] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2: No bootstrap required, opened a new log
I20240501 07:58:32.393566 1568 tablet_bootstrap.cc:492] T 2c969b6e50fd4fd4919c181ae51a47ae P e2c804beb8ef40ecb16c952f8875ca02: No bootstrap required, opened a new log
I20240501 07:58:32.393718 1566 ts_tablet_manager.cc:1367] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 07:58:32.393738 1568 ts_tablet_manager.cc:1367] T 2c969b6e50fd4fd4919c181ae51a47ae P e2c804beb8ef40ecb16c952f8875ca02: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 07:58:32.393932 956 tablet_service.cc:1467] Processing CreateTablet for tablet 2c969b6e50fd4fd4919c181ae51a47ae (DEFAULT_TABLE table=rebalance_test_table_0 [id=cef0e65e05d34b97905a5928033946ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.395030 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c969b6e50fd4fd4919c181ae51a47ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.396286 1567 tablet_bootstrap.cc:654] T 5d48c8225721471f9eea917078d5ef2d P 10aab83192f447829c9b8254e36d94b4: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.397151 1567 log.cc:824] T 5d48c8225721471f9eea917078d5ef2d P 10aab83192f447829c9b8254e36d94b4: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:32.393932 955 tablet_service.cc:1467] Processing CreateTablet for tablet 5d48c8225721471f9eea917078d5ef2d (DEFAULT_TABLE table=rebalance_test_table_0 [id=cef0e65e05d34b97905a5928033946ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.397918 955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d48c8225721471f9eea917078d5ef2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.397859 1568 raft_consensus.cc:348] T 2c969b6e50fd4fd4919c181ae51a47ae P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:32.398084 1568 raft_consensus.cc:374] T 2c969b6e50fd4fd4919c181ae51a47ae P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.398135 1568 raft_consensus.cc:729] T 2c969b6e50fd4fd4919c181ae51a47ae P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.398648 1566 raft_consensus.cc:348] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:32.398854 1566 raft_consensus.cc:374] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.398903 1566 raft_consensus.cc:729] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.398751 1568 consensus_queue.cc:260] T 2c969b6e50fd4fd4919c181ae51a47ae P e2c804beb8ef40ecb16c952f8875ca02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:32.399520 1566 consensus_queue.cc:260] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:32.399921 1568 ts_tablet_manager.cc:1398] T 2c969b6e50fd4fd4919c181ae51a47ae P e2c804beb8ef40ecb16c952f8875ca02: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240501 07:58:32.400236 1568 tablet_bootstrap.cc:492] T c6f89045c20f4ccbb525febd8f5ce56e P e2c804beb8ef40ecb16c952f8875ca02: Bootstrap starting.
I20240501 07:58:32.401279 1568 tablet_bootstrap.cc:654] T c6f89045c20f4ccbb525febd8f5ce56e P e2c804beb8ef40ecb16c952f8875ca02: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.401371 1567 tablet_bootstrap.cc:492] T 5d48c8225721471f9eea917078d5ef2d P 10aab83192f447829c9b8254e36d94b4: No bootstrap required, opened a new log
I20240501 07:58:32.401566 1567 ts_tablet_manager.cc:1367] T 5d48c8225721471f9eea917078d5ef2d P 10aab83192f447829c9b8254e36d94b4: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240501 07:58:32.402566 1568 tablet_bootstrap.cc:492] T c6f89045c20f4ccbb525febd8f5ce56e P e2c804beb8ef40ecb16c952f8875ca02: No bootstrap required, opened a new log
I20240501 07:58:32.402645 1568 ts_tablet_manager.cc:1367] T c6f89045c20f4ccbb525febd8f5ce56e P e2c804beb8ef40ecb16c952f8875ca02: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 07:58:32.403096 1574 tablet_bootstrap.cc:492] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5: Bootstrap starting.
I20240501 07:58:32.403225 1568 raft_consensus.cc:348] T c6f89045c20f4ccbb525febd8f5ce56e P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:32.403322 1568 raft_consensus.cc:374] T c6f89045c20f4ccbb525febd8f5ce56e P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.403349 1568 raft_consensus.cc:729] T c6f89045c20f4ccbb525febd8f5ce56e P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.403465 1568 consensus_queue.cc:260] T c6f89045c20f4ccbb525febd8f5ce56e P e2c804beb8ef40ecb16c952f8875ca02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:32.403725 1568 ts_tablet_manager.cc:1398] T c6f89045c20f4ccbb525febd8f5ce56e P e2c804beb8ef40ecb16c952f8875ca02: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:32.404989 1566 ts_tablet_manager.cc:1398] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240501 07:58:32.405401 1574 tablet_bootstrap.cc:654] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.405781 1567 raft_consensus.cc:348] T 5d48c8225721471f9eea917078d5ef2d P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:32.406021 1567 raft_consensus.cc:374] T 5d48c8225721471f9eea917078d5ef2d P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.406118 1567 raft_consensus.cc:729] T 5d48c8225721471f9eea917078d5ef2d P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10aab83192f447829c9b8254e36d94b4, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.406237 1574 log.cc:824] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:32.406669 1567 consensus_queue.cc:260] T 5d48c8225721471f9eea917078d5ef2d P 10aab83192f447829c9b8254e36d94b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:32.407263 1566 tablet_bootstrap.cc:492] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:32.408607 1567 ts_tablet_manager.cc:1398] T 5d48c8225721471f9eea917078d5ef2d P 10aab83192f447829c9b8254e36d94b4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 07:58:32.409014 1566 tablet_bootstrap.cc:654] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.409073 1574 tablet_bootstrap.cc:492] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5: No bootstrap required, opened a new log
I20240501 07:58:32.409260 1574 ts_tablet_manager.cc:1367] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 07:58:32.409986 1567 tablet_bootstrap.cc:492] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4: Bootstrap starting.
I20240501 07:58:32.410996 1566 tablet_bootstrap.cc:492] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2: No bootstrap required, opened a new log
I20240501 07:58:32.411046 1567 tablet_bootstrap.cc:654] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.411123 1566 ts_tablet_manager.cc:1367] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 07:58:32.411782 1566 raft_consensus.cc:348] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:32.411934 1566 raft_consensus.cc:374] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.411986 1566 raft_consensus.cc:729] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.412148 1566 consensus_queue.cc:260] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:32.412576 1566 ts_tablet_manager.cc:1398] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:32.412808 1566 tablet_bootstrap.cc:492] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:32.413161 1567 tablet_bootstrap.cc:492] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4: No bootstrap required, opened a new log
I20240501 07:58:32.413264 1567 ts_tablet_manager.cc:1367] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:32.413334 1574 raft_consensus.cc:348] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:32.413612 1574 raft_consensus.cc:374] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.413697 1574 raft_consensus.cc:729] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2edc87d834f4e0bb662d501af69f2b5, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.413854 1566 tablet_bootstrap.cc:654] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.413839 1567 raft_consensus.cc:348] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:32.413942 1567 raft_consensus.cc:374] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.413981 1567 raft_consensus.cc:729] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10aab83192f447829c9b8254e36d94b4, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.414113 1567 consensus_queue.cc:260] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:32.414414 1567 ts_tablet_manager.cc:1398] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:32.414650 1574 consensus_queue.cc:260] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
W20240501 07:58:32.415445 630 tablet.cc:2369] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:58:32.416152 1566 tablet_bootstrap.cc:492] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2: No bootstrap required, opened a new log
I20240501 07:58:32.416441 1574 ts_tablet_manager.cc:1398] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240501 07:58:32.416453 1566 ts_tablet_manager.cc:1367] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 07:58:32.416967 1574 tablet_bootstrap.cc:492] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5: Bootstrap starting.
I20240501 07:58:32.416996 1566 raft_consensus.cc:348] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:32.417292 1566 raft_consensus.cc:374] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.417516 1566 raft_consensus.cc:729] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.417675 1566 consensus_queue.cc:260] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:32.418058 1574 tablet_bootstrap.cc:654] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.418188 1566 ts_tablet_manager.cc:1398] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:32.419452 1574 tablet_bootstrap.cc:492] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5: No bootstrap required, opened a new log
I20240501 07:58:32.419546 1574 ts_tablet_manager.cc:1367] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:32.420042 1574 raft_consensus.cc:348] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:32.420138 1574 raft_consensus.cc:374] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.420159 1574 raft_consensus.cc:729] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2edc87d834f4e0bb662d501af69f2b5, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.420300 1574 consensus_queue.cc:260] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:32.420555 1574 ts_tablet_manager.cc:1398] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:32.453104 1581 raft_consensus.cc:482] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:32.453308 1581 raft_consensus.cc:504] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:32.454480 1581 leader_election.cc:288] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475), 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849)
I20240501 07:58:32.459329 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d48c8225721471f9eea917078d5ef2d" candidate_uuid: "e2edc87d834f4e0bb662d501af69f2b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4" is_pre_election: true
I20240501 07:58:32.459332 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d48c8225721471f9eea917078d5ef2d" candidate_uuid: "e2edc87d834f4e0bb662d501af69f2b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" is_pre_election: true
I20240501 07:58:32.459636 845 raft_consensus.cc:2454] T 5d48c8225721471f9eea917078d5ef2d P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2edc87d834f4e0bb662d501af69f2b5 in term 0.
I20240501 07:58:32.459636 583 raft_consensus.cc:2454] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2edc87d834f4e0bb662d501af69f2b5 in term 0.
I20240501 07:58:32.460110 910 leader_election.cc:302] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2, e2edc87d834f4e0bb662d501af69f2b5; no voters:
I20240501 07:58:32.460387 1581 raft_consensus.cc:2781] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:32.460462 1581 raft_consensus.cc:482] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:32.460500 1581 raft_consensus.cc:3037] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.461408 1581 raft_consensus.cc:504] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:32.461752 1581 leader_election.cc:288] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 1 election: Requested vote from peers 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475), 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849)
I20240501 07:58:32.462146 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d48c8225721471f9eea917078d5ef2d" candidate_uuid: "e2edc87d834f4e0bb662d501af69f2b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4"
I20240501 07:58:32.462148 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d48c8225721471f9eea917078d5ef2d" candidate_uuid: "e2edc87d834f4e0bb662d501af69f2b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2"
I20240501 07:58:32.462272 583 raft_consensus.cc:3037] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.462281 845 raft_consensus.cc:3037] T 5d48c8225721471f9eea917078d5ef2d P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.463253 845 raft_consensus.cc:2454] T 5d48c8225721471f9eea917078d5ef2d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2edc87d834f4e0bb662d501af69f2b5 in term 1.
I20240501 07:58:32.463554 909 leader_election.cc:302] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10aab83192f447829c9b8254e36d94b4, e2edc87d834f4e0bb662d501af69f2b5; no voters:
I20240501 07:58:32.463627 583 raft_consensus.cc:2454] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2edc87d834f4e0bb662d501af69f2b5 in term 1.
I20240501 07:58:32.463775 1581 raft_consensus.cc:2781] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:32.463968 1581 raft_consensus.cc:686] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 LEADER]: Becoming Leader. State: Replica: e2edc87d834f4e0bb662d501af69f2b5, State: Running, Role: LEADER
I20240501 07:58:32.464277 1581 consensus_queue.cc:237] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:32.467914 440 catalog_manager.cc:5484] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "e2edc87d834f4e0bb662d501af69f2b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:32.487824 1581 raft_consensus.cc:482] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:32.487978 1581 raft_consensus.cc:504] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:32.488466 1581 leader_election.cc:288] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849), e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577)
I20240501 07:58:32.489094 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c969b6e50fd4fd4919c181ae51a47ae" candidate_uuid: "e2edc87d834f4e0bb662d501af69f2b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" is_pre_election: true
I20240501 07:58:32.489276 583 raft_consensus.cc:2454] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2edc87d834f4e0bb662d501af69f2b5 in term 0.
I20240501 07:58:32.489619 910 leader_election.cc:302] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2, e2edc87d834f4e0bb662d501af69f2b5; no voters:
I20240501 07:58:32.489825 1581 raft_consensus.cc:2781] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:32.489890 1581 raft_consensus.cc:482] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:32.489920 1581 raft_consensus.cc:3037] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.490826 1581 raft_consensus.cc:504] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:32.491228 1581 leader_election.cc:288] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 1 election: Requested vote from peers 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849), e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577)
I20240501 07:58:32.491662 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c969b6e50fd4fd4919c181ae51a47ae" candidate_uuid: "e2edc87d834f4e0bb662d501af69f2b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2"
I20240501 07:58:32.491798 583 raft_consensus.cc:3037] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.492832 583 raft_consensus.cc:2454] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2edc87d834f4e0bb662d501af69f2b5 in term 1.
I20240501 07:58:32.492905 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c969b6e50fd4fd4919c181ae51a47ae" candidate_uuid: "e2edc87d834f4e0bb662d501af69f2b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02" is_pre_election: true
I20240501 07:58:32.492883 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c969b6e50fd4fd4919c181ae51a47ae" candidate_uuid: "e2edc87d834f4e0bb662d501af69f2b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02"
I20240501 07:58:32.493170 910 leader_election.cc:302] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2, e2edc87d834f4e0bb662d501af69f2b5; no voters:
I20240501 07:58:32.493219 714 raft_consensus.cc:2454] T 2c969b6e50fd4fd4919c181ae51a47ae P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2edc87d834f4e0bb662d501af69f2b5 in term 0.
I20240501 07:58:32.493374 1581 raft_consensus.cc:2781] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:32.493463 1581 raft_consensus.cc:686] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 [term 1 LEADER]: Becoming Leader. State: Replica: e2edc87d834f4e0bb662d501af69f2b5, State: Running, Role: LEADER
I20240501 07:58:32.493628 1581 consensus_queue.cc:237] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:32.495702 440 catalog_manager.cc:5484] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "e2edc87d834f4e0bb662d501af69f2b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:32.512998 1023 tablet.cc:2369] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:58:32.564874 892 tablet.cc:2369] T 5d48c8225721471f9eea917078d5ef2d P 10aab83192f447829c9b8254e36d94b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:58:32.592402 1577 raft_consensus.cc:482] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:32.592571 1577 raft_consensus.cc:504] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:32.593664 1577 leader_election.cc:288] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849), e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577)
I20240501 07:58:32.598448 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f89045c20f4ccbb525febd8f5ce56e" candidate_uuid: "10aab83192f447829c9b8254e36d94b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" is_pre_election: true
I20240501 07:58:32.598659 583 raft_consensus.cc:2454] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10aab83192f447829c9b8254e36d94b4 in term 0.
I20240501 07:58:32.598750 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f89045c20f4ccbb525febd8f5ce56e" candidate_uuid: "10aab83192f447829c9b8254e36d94b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02" is_pre_election: true
I20240501 07:58:32.598882 713 raft_consensus.cc:2454] T c6f89045c20f4ccbb525febd8f5ce56e P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10aab83192f447829c9b8254e36d94b4 in term 0.
I20240501 07:58:32.599076 779 leader_election.cc:302] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10aab83192f447829c9b8254e36d94b4, 2c148dc68b5a4a38ba10be4e1e4b8cd2; no voters:
I20240501 07:58:32.599372 1577 raft_consensus.cc:2781] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:32.599465 1577 raft_consensus.cc:482] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:32.599495 1577 raft_consensus.cc:3037] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.600570 1577 raft_consensus.cc:504] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:32.600956 1577 leader_election.cc:288] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 1 election: Requested vote from peers 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849), e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577)
I20240501 07:58:32.601437 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f89045c20f4ccbb525febd8f5ce56e" candidate_uuid: "10aab83192f447829c9b8254e36d94b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2"
I20240501 07:58:32.601437 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f89045c20f4ccbb525febd8f5ce56e" candidate_uuid: "10aab83192f447829c9b8254e36d94b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02"
I20240501 07:58:32.601569 583 raft_consensus.cc:3037] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.601575 713 raft_consensus.cc:3037] T c6f89045c20f4ccbb525febd8f5ce56e P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.602610 713 raft_consensus.cc:2454] T c6f89045c20f4ccbb525febd8f5ce56e P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10aab83192f447829c9b8254e36d94b4 in term 1.
I20240501 07:58:32.602610 583 raft_consensus.cc:2454] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10aab83192f447829c9b8254e36d94b4 in term 1.
I20240501 07:58:32.602939 779 leader_election.cc:302] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10aab83192f447829c9b8254e36d94b4, 2c148dc68b5a4a38ba10be4e1e4b8cd2; no voters:
I20240501 07:58:32.603118 1577 raft_consensus.cc:2781] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:32.603392 1577 raft_consensus.cc:686] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 [term 1 LEADER]: Becoming Leader. State: Replica: 10aab83192f447829c9b8254e36d94b4, State: Running, Role: LEADER
I20240501 07:58:32.603765 1577 consensus_queue.cc:237] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:32.607051 440 catalog_manager.cc:5484] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10aab83192f447829c9b8254e36d94b4 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "10aab83192f447829c9b8254e36d94b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:32.608835 761 tablet.cc:2369] T 2c969b6e50fd4fd4919c181ae51a47ae P e2c804beb8ef40ecb16c952f8875ca02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:58:32.731408 1592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:32.731684 1592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:32.735512 1592 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 07:58:32.844959 583 raft_consensus.cc:1261] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:32.845176 845 raft_consensus.cc:1261] T 5d48c8225721471f9eea917078d5ef2d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:32.845722 1621 consensus_queue.cc:1035] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:32.846175 713 raft_consensus.cc:3037] T 2c969b6e50fd4fd4919c181ae51a47ae P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:32.847532 583 raft_consensus.cc:1261] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Refusing update from remote peer 10aab83192f447829c9b8254e36d94b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:32.847621 714 raft_consensus.cc:1261] T c6f89045c20f4ccbb525febd8f5ce56e P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Refusing update from remote peer 10aab83192f447829c9b8254e36d94b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:32.847836 1581 consensus_queue.cc:1035] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:32.848170 582 raft_consensus.cc:1261] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:32.849246 713 raft_consensus.cc:1261] T 2c969b6e50fd4fd4919c181ae51a47ae P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:32.849619 1621 consensus_queue.cc:1035] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:32.849793 1581 consensus_queue.cc:1035] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:32.849782 1577 consensus_queue.cc:1035] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:32.850208 1577 consensus_queue.cc:1035] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:32.859771 1620 mvcc.cc:205] Tried to move back new op lower bound from 7022798081401344000 to 7022798079856332800. Current Snapshot: MvccSnapshot[applied={T|T < 7022798081401344000}]
I20240501 07:58:32.873492 1617 mvcc.cc:205] Tried to move back new op lower bound from 7022798081496465408 to 7022798079856332800. Current Snapshot: MvccSnapshot[applied={T|T < 7022798081401344000}]
I20240501 07:58:32.877730 1626 mvcc.cc:205] Tried to move back new op lower bound from 7022798081415987200 to 7022798080427868160. Current Snapshot: MvccSnapshot[applied={T|T < 7022798081415987200}]
I20240501 07:58:32.899951 1631 mvcc.cc:205] Tried to move back new op lower bound from 7022798081401409536 to 7022798079976067072. Current Snapshot: MvccSnapshot[applied={T|T < 7022798081401409536}]
INSERT report
rows total: 2000
time total: 149.033 ms
time per row: 0.0745165 ms
I20240501 07:58:32.941125 439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35336:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:32.947467 694 tablet_service.cc:1467] Processing CreateTablet for tablet e0638f250a164460a694cd41b3e9f27d (DEFAULT_TABLE table=rebalance_test_table_1 [id=e08d5aefd3334c55a539f0f47df899e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.947525 956 tablet_service.cc:1467] Processing CreateTablet for tablet e0638f250a164460a694cd41b3e9f27d (DEFAULT_TABLE table=rebalance_test_table_1 [id=e08d5aefd3334c55a539f0f47df899e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.947831 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0638f250a164460a694cd41b3e9f27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.947831 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0638f250a164460a694cd41b3e9f27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.948772 692 tablet_service.cc:1467] Processing CreateTablet for tablet e7eefec740384c688dc6e149b5ea2769 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e08d5aefd3334c55a539f0f47df899e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.949065 692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7eefec740384c688dc6e149b5ea2769. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.949071 563 tablet_service.cc:1467] Processing CreateTablet for tablet e7eefec740384c688dc6e149b5ea2769 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e08d5aefd3334c55a539f0f47df899e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.949333 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7eefec740384c688dc6e149b5ea2769. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.950285 693 tablet_service.cc:1467] Processing CreateTablet for tablet b8e9192810694fa1bf54ca88fb619d11 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e08d5aefd3334c55a539f0f47df899e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.950559 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8e9192810694fa1bf54ca88fb619d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.951092 956 tablet_service.cc:1467] Processing CreateTablet for tablet b8e9192810694fa1bf54ca88fb619d11 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e08d5aefd3334c55a539f0f47df899e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.951324 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8e9192810694fa1bf54ca88fb619d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.951656 1653 tablet_bootstrap.cc:492] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:32.952658 1653 tablet_bootstrap.cc:654] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.952972 825 tablet_service.cc:1467] Processing CreateTablet for tablet e0638f250a164460a694cd41b3e9f27d (DEFAULT_TABLE table=rebalance_test_table_1 [id=e08d5aefd3334c55a539f0f47df899e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.953233 825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0638f250a164460a694cd41b3e9f27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.953846 1653 tablet_bootstrap.cc:492] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: No bootstrap required, opened a new log
I20240501 07:58:32.953913 1652 tablet_bootstrap.cc:492] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5: Bootstrap starting.
I20240501 07:58:32.953924 1653 ts_tablet_manager.cc:1367] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:32.954739 1652 tablet_bootstrap.cc:654] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.954860 1653 raft_consensus.cc:348] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:32.954989 1653 raft_consensus.cc:374] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.955026 1653 raft_consensus.cc:729] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.955214 1653 consensus_queue.cc:260] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:32.955307 1651 tablet_bootstrap.cc:492] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02: Bootstrap starting.
I20240501 07:58:32.955549 1653 ts_tablet_manager.cc:1398] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:32.956179 1651 tablet_bootstrap.cc:654] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.956218 1656 tablet_bootstrap.cc:492] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4: Bootstrap starting.
I20240501 07:58:32.956455 824 tablet_service.cc:1467] Processing CreateTablet for tablet b8e9192810694fa1bf54ca88fb619d11 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e08d5aefd3334c55a539f0f47df899e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.956694 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8e9192810694fa1bf54ca88fb619d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.957067 1656 tablet_bootstrap.cc:654] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.957121 1652 tablet_bootstrap.cc:492] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5: No bootstrap required, opened a new log
I20240501 07:58:32.957204 1652 ts_tablet_manager.cc:1367] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:32.957659 1652 raft_consensus.cc:348] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:32.957747 1652 raft_consensus.cc:374] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.957774 1652 raft_consensus.cc:729] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2edc87d834f4e0bb662d501af69f2b5, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.957903 1652 consensus_queue.cc:260] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:32.958137 1656 tablet_bootstrap.cc:492] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4: No bootstrap required, opened a new log
I20240501 07:58:32.958173 1652 ts_tablet_manager.cc:1398] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:32.958212 1656 ts_tablet_manager.cc:1367] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:32.958673 1656 raft_consensus.cc:348] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:32.958786 1656 raft_consensus.cc:374] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.958822 1656 raft_consensus.cc:729] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10aab83192f447829c9b8254e36d94b4, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.958930 1651 tablet_bootstrap.cc:492] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02: No bootstrap required, opened a new log
I20240501 07:58:32.958948 1656 consensus_queue.cc:260] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:32.959013 1651 ts_tablet_manager.cc:1367] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:58:32.959203 1656 ts_tablet_manager.cc:1398] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:32.959846 1651 raft_consensus.cc:348] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:32.959933 1651 raft_consensus.cc:374] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.959954 1651 raft_consensus.cc:729] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.960062 1651 consensus_queue.cc:260] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:32.960309 1651 ts_tablet_manager.cc:1398] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:32.960433 1651 tablet_bootstrap.cc:492] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02: Bootstrap starting.
I20240501 07:58:32.961306 1651 tablet_bootstrap.cc:654] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.961882 1652 tablet_bootstrap.cc:492] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5: Bootstrap starting.
I20240501 07:58:32.962790 1652 tablet_bootstrap.cc:654] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.963789 1651 tablet_bootstrap.cc:492] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02: No bootstrap required, opened a new log
I20240501 07:58:32.963881 1651 ts_tablet_manager.cc:1367] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:32.964272 1651 raft_consensus.cc:348] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:32.964360 1651 raft_consensus.cc:374] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.964380 1651 raft_consensus.cc:729] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.964514 1651 consensus_queue.cc:260] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:32.964751 1651 ts_tablet_manager.cc:1398] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:32.964948 1651 tablet_bootstrap.cc:492] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02: Bootstrap starting.
I20240501 07:58:32.964972 823 tablet_service.cc:1467] Processing CreateTablet for tablet e7eefec740384c688dc6e149b5ea2769 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e08d5aefd3334c55a539f0f47df899e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:32.965212 1652 tablet_bootstrap.cc:492] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5: No bootstrap required, opened a new log
I20240501 07:58:32.965306 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7eefec740384c688dc6e149b5ea2769. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:32.965312 1652 ts_tablet_manager.cc:1367] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:32.965775 1652 raft_consensus.cc:348] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:32.965878 1652 raft_consensus.cc:374] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.965963 1652 raft_consensus.cc:729] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2edc87d834f4e0bb662d501af69f2b5, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.966099 1651 tablet_bootstrap.cc:654] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.966126 1652 consensus_queue.cc:260] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:32.966596 1652 ts_tablet_manager.cc:1398] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:32.967645 1656 tablet_bootstrap.cc:492] T e7eefec740384c688dc6e149b5ea2769 P 10aab83192f447829c9b8254e36d94b4: Bootstrap starting.
I20240501 07:58:32.968611 1656 tablet_bootstrap.cc:654] T e7eefec740384c688dc6e149b5ea2769 P 10aab83192f447829c9b8254e36d94b4: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.970487 1656 tablet_bootstrap.cc:492] T e7eefec740384c688dc6e149b5ea2769 P 10aab83192f447829c9b8254e36d94b4: No bootstrap required, opened a new log
I20240501 07:58:32.970625 1656 ts_tablet_manager.cc:1367] T e7eefec740384c688dc6e149b5ea2769 P 10aab83192f447829c9b8254e36d94b4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:32.971158 1651 tablet_bootstrap.cc:492] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02: No bootstrap required, opened a new log
I20240501 07:58:32.971273 1651 ts_tablet_manager.cc:1367] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 07:58:32.971113 1656 raft_consensus.cc:348] T e7eefec740384c688dc6e149b5ea2769 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:32.971534 1656 raft_consensus.cc:374] T e7eefec740384c688dc6e149b5ea2769 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.971618 1656 raft_consensus.cc:729] T e7eefec740384c688dc6e149b5ea2769 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10aab83192f447829c9b8254e36d94b4, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.971823 1656 consensus_queue.cc:260] T e7eefec740384c688dc6e149b5ea2769 P 10aab83192f447829c9b8254e36d94b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:32.972200 1656 ts_tablet_manager.cc:1398] T e7eefec740384c688dc6e149b5ea2769 P 10aab83192f447829c9b8254e36d94b4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:32.972131 1651 raft_consensus.cc:348] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:32.972397 1656 tablet_bootstrap.cc:492] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4: Bootstrap starting.
I20240501 07:58:32.972643 1651 raft_consensus.cc:374] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.972712 1651 raft_consensus.cc:729] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.972956 1651 consensus_queue.cc:260] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:32.973240 1651 ts_tablet_manager.cc:1398] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:32.973444 1656 tablet_bootstrap.cc:654] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:32.975440 1656 tablet_bootstrap.cc:492] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4: No bootstrap required, opened a new log
I20240501 07:58:32.975585 1656 ts_tablet_manager.cc:1367] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:32.976051 1656 raft_consensus.cc:348] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:32.976166 1656 raft_consensus.cc:374] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:32.976197 1656 raft_consensus.cc:729] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10aab83192f447829c9b8254e36d94b4, State: Initialized, Role: FOLLOWER
I20240501 07:58:32.976336 1656 consensus_queue.cc:260] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:32.976642 1656 ts_tablet_manager.cc:1398] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:33.082530 1572 raft_consensus.cc:482] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:33.082746 1572 raft_consensus.cc:504] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:33.083846 1572 leader_election.cc:288] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475), e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973)
I20240501 07:58:33.089010 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e9192810694fa1bf54ca88fb619d11" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4" is_pre_election: true
I20240501 07:58:33.089059 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e9192810694fa1bf54ca88fb619d11" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5" is_pre_election: true
I20240501 07:58:33.089210 845 raft_consensus.cc:2454] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 0.
I20240501 07:58:33.089257 976 raft_consensus.cc:2454] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 0.
I20240501 07:58:33.089669 647 leader_election.cc:302] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10aab83192f447829c9b8254e36d94b4, e2c804beb8ef40ecb16c952f8875ca02, e2edc87d834f4e0bb662d501af69f2b5; no voters:
I20240501 07:58:33.089934 1572 raft_consensus.cc:2781] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:33.089987 1572 raft_consensus.cc:482] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:33.090023 1572 raft_consensus.cc:3037] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.130678 1572 raft_consensus.cc:504] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:33.131297 1572 leader_election.cc:288] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 election: Requested vote from peers 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475), e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973)
I20240501 07:58:33.131877 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e9192810694fa1bf54ca88fb619d11" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5"
I20240501 07:58:33.132021 976 raft_consensus.cc:3037] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.131956 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e9192810694fa1bf54ca88fb619d11" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4"
I20240501 07:58:33.132164 845 raft_consensus.cc:3037] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.133096 976 raft_consensus.cc:2454] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 1.
I20240501 07:58:33.133323 845 raft_consensus.cc:2454] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 1.
I20240501 07:58:33.133491 649 leader_election.cc:302] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2c804beb8ef40ecb16c952f8875ca02, e2edc87d834f4e0bb662d501af69f2b5; no voters:
I20240501 07:58:33.133883 1572 raft_consensus.cc:2781] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:33.134099 1572 raft_consensus.cc:686] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Becoming Leader. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Running, Role: LEADER
I20240501 07:58:33.134467 1572 consensus_queue.cc:237] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:33.137053 439 catalog_manager.cc:5484] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:33.176891 1646 raft_consensus.cc:482] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:33.177064 1646 raft_consensus.cc:504] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:33.177574 1646 leader_election.cc:288] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577), 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475)
I20240501 07:58:33.178083 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0638f250a164460a694cd41b3e9f27d" candidate_uuid: "e2edc87d834f4e0bb662d501af69f2b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02" is_pre_election: true
I20240501 07:58:33.178098 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0638f250a164460a694cd41b3e9f27d" candidate_uuid: "e2edc87d834f4e0bb662d501af69f2b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4" is_pre_election: true
I20240501 07:58:33.178241 713 raft_consensus.cc:2454] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2edc87d834f4e0bb662d501af69f2b5 in term 0.
I20240501 07:58:33.178259 845 raft_consensus.cc:2454] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2edc87d834f4e0bb662d501af69f2b5 in term 0.
I20240501 07:58:33.178565 910 leader_election.cc:302] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2c804beb8ef40ecb16c952f8875ca02, e2edc87d834f4e0bb662d501af69f2b5; no voters:
I20240501 07:58:33.178777 1646 raft_consensus.cc:2781] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:33.178826 1646 raft_consensus.cc:482] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:33.178862 1646 raft_consensus.cc:3037] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.179738 1646 raft_consensus.cc:504] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:33.180102 1646 leader_election.cc:288] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 1 election: Requested vote from peers e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577), 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475)
I20240501 07:58:33.180560 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0638f250a164460a694cd41b3e9f27d" candidate_uuid: "e2edc87d834f4e0bb662d501af69f2b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02"
I20240501 07:58:33.180737 713 raft_consensus.cc:3037] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.181526 713 raft_consensus.cc:2454] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2edc87d834f4e0bb662d501af69f2b5 in term 1.
I20240501 07:58:33.181833 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0638f250a164460a694cd41b3e9f27d" candidate_uuid: "e2edc87d834f4e0bb662d501af69f2b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4"
I20240501 07:58:33.181905 910 leader_election.cc:302] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2c804beb8ef40ecb16c952f8875ca02, e2edc87d834f4e0bb662d501af69f2b5; no voters:
I20240501 07:58:33.181977 845 raft_consensus.cc:3037] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.182077 1646 raft_consensus.cc:2781] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:33.182160 1646 raft_consensus.cc:686] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 LEADER]: Becoming Leader. State: Replica: e2edc87d834f4e0bb662d501af69f2b5, State: Running, Role: LEADER
I20240501 07:58:33.182327 1646 consensus_queue.cc:237] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:33.183043 845 raft_consensus.cc:2454] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2edc87d834f4e0bb662d501af69f2b5 in term 1.
I20240501 07:58:33.184324 439 catalog_manager.cc:5484] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "e2edc87d834f4e0bb662d501af69f2b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:33.260915 1572 raft_consensus.cc:482] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:33.261083 1572 raft_consensus.cc:504] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:33.261611 1572 leader_election.cc:288] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849), 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475)
I20240501 07:58:33.262228 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7eefec740384c688dc6e149b5ea2769" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4" is_pre_election: true
I20240501 07:58:33.262391 845 raft_consensus.cc:2454] T e7eefec740384c688dc6e149b5ea2769 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 0.
I20240501 07:58:33.262734 647 leader_election.cc:302] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10aab83192f447829c9b8254e36d94b4, e2c804beb8ef40ecb16c952f8875ca02; no voters:
I20240501 07:58:33.262913 1572 raft_consensus.cc:2781] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:33.262966 1572 raft_consensus.cc:482] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:33.263002 1572 raft_consensus.cc:3037] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.264008 1572 raft_consensus.cc:504] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:33.264400 1572 leader_election.cc:288] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 election: Requested vote from peers 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849), 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475)
I20240501 07:58:33.264832 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7eefec740384c688dc6e149b5ea2769" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4"
I20240501 07:58:33.265033 845 raft_consensus.cc:3037] T e7eefec740384c688dc6e149b5ea2769 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.265630 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7eefec740384c688dc6e149b5ea2769" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" is_pre_election: true
I20240501 07:58:33.265776 582 raft_consensus.cc:2454] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 0.
I20240501 07:58:33.265990 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7eefec740384c688dc6e149b5ea2769" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2"
I20240501 07:58:33.266033 845 raft_consensus.cc:2454] T e7eefec740384c688dc6e149b5ea2769 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 1.
I20240501 07:58:33.266089 581 raft_consensus.cc:3037] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.266355 647 leader_election.cc:302] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10aab83192f447829c9b8254e36d94b4, e2c804beb8ef40ecb16c952f8875ca02; no voters:
I20240501 07:58:33.266552 1572 raft_consensus.cc:2781] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:33.266635 1572 raft_consensus.cc:686] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Becoming Leader. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Running, Role: LEADER
I20240501 07:58:33.266782 1572 consensus_queue.cc:237] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:33.267061 581 raft_consensus.cc:2454] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 1.
I20240501 07:58:33.269078 439 catalog_manager.cc:5484] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:33.464816 1671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:33.465109 1671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:33.468581 1671 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 07:58:33.575706 845 raft_consensus.cc:1261] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:33.576516 1646 consensus_queue.cc:1035] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:33.577023 581 raft_consensus.cc:1261] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:33.577314 1666 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:33.579413 1572 consensus_queue.cc:1035] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:33.579509 845 raft_consensus.cc:1261] T e7eefec740384c688dc6e149b5ea2769 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:33.579895 1666 consensus_queue.cc:1035] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:33.580718 976 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:33.581581 1666 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:33.587534 713 raft_consensus.cc:1261] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:33.588182 1646 consensus_queue.cc:1035] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 119.536 ms
time per row: 0.0597681 ms
I20240501 07:58:33.642792 441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35336:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:33.649664 563 tablet_service.cc:1467] Processing CreateTablet for tablet 9d5700ab4d4f465bbe08ad25d0b9084f (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f6bea7fb1b842af87d7bafed90d8fb6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.650044 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d5700ab4d4f465bbe08ad25d0b9084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.650231 561 tablet_service.cc:1467] Processing CreateTablet for tablet 88887f8c213f46b2b4d4555d6a13c346 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f6bea7fb1b842af87d7bafed90d8fb6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.651015 955 tablet_service.cc:1467] Processing CreateTablet for tablet 88887f8c213f46b2b4d4555d6a13c346 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f6bea7fb1b842af87d7bafed90d8fb6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.651015 956 tablet_service.cc:1467] Processing CreateTablet for tablet 9d5700ab4d4f465bbe08ad25d0b9084f (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f6bea7fb1b842af87d7bafed90d8fb6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.651405 955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88887f8c213f46b2b4d4555d6a13c346. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.651588 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d5700ab4d4f465bbe08ad25d0b9084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.651988 562 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab169c1536c4b849918c7cd593048c1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f6bea7fb1b842af87d7bafed90d8fb6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.652244 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab169c1536c4b849918c7cd593048c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.652235 954 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab169c1536c4b849918c7cd593048c1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f6bea7fb1b842af87d7bafed90d8fb6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.652475 954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab169c1536c4b849918c7cd593048c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.652783 693 tablet_service.cc:1467] Processing CreateTablet for tablet 9d5700ab4d4f465bbe08ad25d0b9084f (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f6bea7fb1b842af87d7bafed90d8fb6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.653051 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d5700ab4d4f465bbe08ad25d0b9084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.654668 824 tablet_service.cc:1467] Processing CreateTablet for tablet 88887f8c213f46b2b4d4555d6a13c346 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f6bea7fb1b842af87d7bafed90d8fb6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.654821 692 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab169c1536c4b849918c7cd593048c1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f6bea7fb1b842af87d7bafed90d8fb6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:33.654973 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88887f8c213f46b2b4d4555d6a13c346. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.655050 692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab169c1536c4b849918c7cd593048c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.656352 1734 tablet_bootstrap.cc:492] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:33.656385 1736 tablet_bootstrap.cc:492] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02: Bootstrap starting.
I20240501 07:58:33.656518 1735 tablet_bootstrap.cc:492] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5: Bootstrap starting.
I20240501 07:58:33.657167 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88887f8c213f46b2b4d4555d6a13c346. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.657430 1734 tablet_bootstrap.cc:654] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.657608 1735 tablet_bootstrap.cc:654] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.658704 1739 tablet_bootstrap.cc:492] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4: Bootstrap starting.
I20240501 07:58:33.659583 1739 tablet_bootstrap.cc:654] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.660105 1736 tablet_bootstrap.cc:654] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.660324 1735 tablet_bootstrap.cc:492] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5: No bootstrap required, opened a new log
I20240501 07:58:33.660508 1734 tablet_bootstrap.cc:492] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2: No bootstrap required, opened a new log
I20240501 07:58:33.660643 1734 ts_tablet_manager.cc:1367] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:33.660851 1735 ts_tablet_manager.cc:1367] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:33.661212 1734 raft_consensus.cc:348] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:33.661376 1736 tablet_bootstrap.cc:492] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02: No bootstrap required, opened a new log
I20240501 07:58:33.661489 1736 ts_tablet_manager.cc:1367] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:33.661548 1734 raft_consensus.cc:374] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.661607 1734 raft_consensus.cc:729] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.661769 1734 consensus_queue.cc:260] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:33.662122 1734 ts_tablet_manager.cc:1398] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:33.662392 1734 tablet_bootstrap.cc:492] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:33.662361 1736 raft_consensus.cc:348] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:33.662623 1736 raft_consensus.cc:374] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.661785 1735 raft_consensus.cc:348] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:33.662755 1736 raft_consensus.cc:729] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.662855 1735 raft_consensus.cc:374] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.662992 1736 consensus_queue.cc:260] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:33.663455 1736 ts_tablet_manager.cc:1398] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:33.663714 1736 tablet_bootstrap.cc:492] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02: Bootstrap starting.
I20240501 07:58:33.664489 1734 tablet_bootstrap.cc:654] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.664855 1736 tablet_bootstrap.cc:654] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.666548 1736 tablet_bootstrap.cc:492] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02: No bootstrap required, opened a new log
I20240501 07:58:33.666731 1736 ts_tablet_manager.cc:1367] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:33.663852 1735 raft_consensus.cc:729] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2edc87d834f4e0bb662d501af69f2b5, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.666956 1734 tablet_bootstrap.cc:492] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: No bootstrap required, opened a new log
I20240501 07:58:33.667043 1734 ts_tablet_manager.cc:1367] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 07:58:33.667261 1735 consensus_queue.cc:260] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:33.667325 1736 raft_consensus.cc:348] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:33.667418 1736 raft_consensus.cc:374] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.667451 1736 raft_consensus.cc:729] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.667570 1734 raft_consensus.cc:348] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:33.667712 1734 raft_consensus.cc:374] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.667764 1734 raft_consensus.cc:729] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.667906 1734 consensus_queue.cc:260] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:33.668116 1735 ts_tablet_manager.cc:1398] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:33.667631 1736 consensus_queue.cc:260] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:33.668468 1736 ts_tablet_manager.cc:1398] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:33.668601 1734 ts_tablet_manager.cc:1398] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:33.668787 1734 tablet_bootstrap.cc:492] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:33.668788 1735 tablet_bootstrap.cc:492] T 5ab169c1536c4b849918c7cd593048c1 P e2edc87d834f4e0bb662d501af69f2b5: Bootstrap starting.
I20240501 07:58:33.669088 1739 tablet_bootstrap.cc:492] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4: No bootstrap required, opened a new log
I20240501 07:58:33.669222 1739 ts_tablet_manager.cc:1367] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 07:58:33.669839 1734 tablet_bootstrap.cc:654] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.669853 1735 tablet_bootstrap.cc:654] T 5ab169c1536c4b849918c7cd593048c1 P e2edc87d834f4e0bb662d501af69f2b5: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.669771 1739 raft_consensus.cc:348] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:33.669939 1739 raft_consensus.cc:374] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.669992 1739 raft_consensus.cc:729] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10aab83192f447829c9b8254e36d94b4, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.670171 1739 consensus_queue.cc:260] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:33.671202 1739 ts_tablet_manager.cc:1398] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:33.671732 1735 tablet_bootstrap.cc:492] T 5ab169c1536c4b849918c7cd593048c1 P e2edc87d834f4e0bb662d501af69f2b5: No bootstrap required, opened a new log
I20240501 07:58:33.671834 1735 ts_tablet_manager.cc:1367] T 5ab169c1536c4b849918c7cd593048c1 P e2edc87d834f4e0bb662d501af69f2b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:33.672372 1735 raft_consensus.cc:348] T 5ab169c1536c4b849918c7cd593048c1 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:33.672495 1735 raft_consensus.cc:374] T 5ab169c1536c4b849918c7cd593048c1 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.672530 1735 raft_consensus.cc:729] T 5ab169c1536c4b849918c7cd593048c1 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2edc87d834f4e0bb662d501af69f2b5, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.672657 1735 consensus_queue.cc:260] T 5ab169c1536c4b849918c7cd593048c1 P e2edc87d834f4e0bb662d501af69f2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:33.672953 1735 ts_tablet_manager.cc:1398] T 5ab169c1536c4b849918c7cd593048c1 P e2edc87d834f4e0bb662d501af69f2b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:33.673139 1735 tablet_bootstrap.cc:492] T 88887f8c213f46b2b4d4555d6a13c346 P e2edc87d834f4e0bb662d501af69f2b5: Bootstrap starting.
I20240501 07:58:33.673472 1734 tablet_bootstrap.cc:492] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: No bootstrap required, opened a new log
I20240501 07:58:33.673583 1734 ts_tablet_manager.cc:1367] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 07:58:33.674002 1734 raft_consensus.cc:348] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:33.674106 1734 raft_consensus.cc:374] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.674139 1734 raft_consensus.cc:729] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.674192 1735 tablet_bootstrap.cc:654] T 88887f8c213f46b2b4d4555d6a13c346 P e2edc87d834f4e0bb662d501af69f2b5: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.674273 1734 consensus_queue.cc:260] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:33.674594 1734 ts_tablet_manager.cc:1398] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:33.675880 1735 tablet_bootstrap.cc:492] T 88887f8c213f46b2b4d4555d6a13c346 P e2edc87d834f4e0bb662d501af69f2b5: No bootstrap required, opened a new log
I20240501 07:58:33.676025 1735 ts_tablet_manager.cc:1367] T 88887f8c213f46b2b4d4555d6a13c346 P e2edc87d834f4e0bb662d501af69f2b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:33.676841 1735 raft_consensus.cc:348] T 88887f8c213f46b2b4d4555d6a13c346 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:33.676981 1735 raft_consensus.cc:374] T 88887f8c213f46b2b4d4555d6a13c346 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.677012 1735 raft_consensus.cc:729] T 88887f8c213f46b2b4d4555d6a13c346 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2edc87d834f4e0bb662d501af69f2b5, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.677249 1735 consensus_queue.cc:260] T 88887f8c213f46b2b4d4555d6a13c346 P e2edc87d834f4e0bb662d501af69f2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:33.677774 1735 ts_tablet_manager.cc:1398] T 88887f8c213f46b2b4d4555d6a13c346 P e2edc87d834f4e0bb662d501af69f2b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 07:58:33.683156 1720 raft_consensus.cc:482] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:33.683329 1720 raft_consensus.cc:504] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:33.684515 1720 leader_election.cc:288] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577), e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973)
I20240501 07:58:33.689019 1725 raft_consensus.cc:482] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:33.689177 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5700ab4d4f465bbe08ad25d0b9084f" candidate_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02" is_pre_election: true
I20240501 07:58:33.689106 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5700ab4d4f465bbe08ad25d0b9084f" candidate_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5" is_pre_election: true
I20240501 07:58:33.689160 1725 raft_consensus.cc:504] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:33.689306 976 raft_consensus.cc:2454] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c148dc68b5a4a38ba10be4e1e4b8cd2 in term 0.
I20240501 07:58:33.689425 713 raft_consensus.cc:2454] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c148dc68b5a4a38ba10be4e1e4b8cd2 in term 0.
I20240501 07:58:33.689877 518 leader_election.cc:302] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2, e2edc87d834f4e0bb662d501af69f2b5; no voters:
I20240501 07:58:33.689950 1725 leader_election.cc:288] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849), e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973)
I20240501 07:58:33.690356 1720 raft_consensus.cc:2781] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:33.690474 1720 raft_consensus.cc:482] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:33.690529 1720 raft_consensus.cc:3037] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.690563 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5700ab4d4f465bbe08ad25d0b9084f" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" is_pre_election: true
I20240501 07:58:33.690564 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5700ab4d4f465bbe08ad25d0b9084f" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5" is_pre_election: true
I20240501 07:58:33.690761 976 raft_consensus.cc:2454] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 0.
I20240501 07:58:33.691186 649 leader_election.cc:302] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2c804beb8ef40ecb16c952f8875ca02, e2edc87d834f4e0bb662d501af69f2b5; no voters:
I20240501 07:58:33.691377 1725 raft_consensus.cc:2781] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:33.691529 1725 raft_consensus.cc:482] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:33.691565 1725 raft_consensus.cc:3037] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.691658 1720 raft_consensus.cc:504] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:33.691931 581 raft_consensus.cc:2379] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2c804beb8ef40ecb16c952f8875ca02 in current term 1: Already voted for candidate 2c148dc68b5a4a38ba10be4e1e4b8cd2 in this term.
I20240501 07:58:33.692102 1720 leader_election.cc:288] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [CANDIDATE]: Term 1 election: Requested vote from peers e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577), e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973)
I20240501 07:58:33.692807 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5700ab4d4f465bbe08ad25d0b9084f" candidate_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5"
I20240501 07:58:33.692916 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5700ab4d4f465bbe08ad25d0b9084f" candidate_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02"
I20240501 07:58:33.693001 976 raft_consensus.cc:3037] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.693233 1725 raft_consensus.cc:504] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:33.693564 713 raft_consensus.cc:2379] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2c148dc68b5a4a38ba10be4e1e4b8cd2 in current term 1: Already voted for candidate e2c804beb8ef40ecb16c952f8875ca02 in this term.
I20240501 07:58:33.693724 1725 leader_election.cc:288] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 election: Requested vote from peers 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849), e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973)
I20240501 07:58:33.694188 976 raft_consensus.cc:2454] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c148dc68b5a4a38ba10be4e1e4b8cd2 in term 1.
I20240501 07:58:33.694212 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5700ab4d4f465bbe08ad25d0b9084f" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2"
I20240501 07:58:33.694361 581 raft_consensus.cc:2379] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e2c804beb8ef40ecb16c952f8875ca02 in current term 1: Already voted for candidate 2c148dc68b5a4a38ba10be4e1e4b8cd2 in this term.
I20240501 07:58:33.694551 518 leader_election.cc:302] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2, e2edc87d834f4e0bb662d501af69f2b5; no voters: e2c804beb8ef40ecb16c952f8875ca02
I20240501 07:58:33.694723 1720 raft_consensus.cc:2781] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:33.694813 1720 raft_consensus.cc:686] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 LEADER]: Becoming Leader. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Running, Role: LEADER
I20240501 07:58:33.694898 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5700ab4d4f465bbe08ad25d0b9084f" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5"
I20240501 07:58:33.695055 976 raft_consensus.cc:2379] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e2c804beb8ef40ecb16c952f8875ca02 in current term 1: Already voted for candidate 2c148dc68b5a4a38ba10be4e1e4b8cd2 in this term.
I20240501 07:58:33.695102 1720 consensus_queue.cc:237] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:33.695406 649 leader_election.cc:302] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2c804beb8ef40ecb16c952f8875ca02; no voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2, e2edc87d834f4e0bb662d501af69f2b5
I20240501 07:58:33.695647 1725 raft_consensus.cc:2726] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 07:58:33.698030 439 catalog_manager.cc:5484] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:33.839807 1725 raft_consensus.cc:482] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:33.839973 1725 raft_consensus.cc:504] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:33.840415 1725 leader_election.cc:288] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973), 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849)
I20240501 07:58:33.840947 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab169c1536c4b849918c7cd593048c1" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" is_pre_election: true
I20240501 07:58:33.841017 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab169c1536c4b849918c7cd593048c1" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5" is_pre_election: true
I20240501 07:58:33.841092 581 raft_consensus.cc:2454] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 0.
I20240501 07:58:33.841156 976 raft_consensus.cc:2454] T 5ab169c1536c4b849918c7cd593048c1 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 0.
I20240501 07:58:33.841470 649 leader_election.cc:302] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2c804beb8ef40ecb16c952f8875ca02, e2edc87d834f4e0bb662d501af69f2b5; no voters:
I20240501 07:58:33.841768 1725 raft_consensus.cc:2781] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:33.841871 1725 raft_consensus.cc:482] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:33.841912 1725 raft_consensus.cc:3037] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.842944 1725 raft_consensus.cc:504] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:33.843329 1725 leader_election.cc:288] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 election: Requested vote from peers e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973), 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849)
I20240501 07:58:33.843788 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab169c1536c4b849918c7cd593048c1" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5"
I20240501 07:58:33.843814 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab169c1536c4b849918c7cd593048c1" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2"
I20240501 07:58:33.843904 976 raft_consensus.cc:3037] T 5ab169c1536c4b849918c7cd593048c1 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.843923 581 raft_consensus.cc:3037] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.844696 976 raft_consensus.cc:2454] T 5ab169c1536c4b849918c7cd593048c1 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 1.
I20240501 07:58:33.844705 581 raft_consensus.cc:2454] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 1.
I20240501 07:58:33.845044 649 leader_election.cc:302] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2, e2c804beb8ef40ecb16c952f8875ca02, e2edc87d834f4e0bb662d501af69f2b5; no voters:
I20240501 07:58:33.845263 1725 raft_consensus.cc:2781] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:33.845346 1725 raft_consensus.cc:686] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Becoming Leader. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Running, Role: LEADER
I20240501 07:58:33.845527 1725 consensus_queue.cc:237] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:33.847434 439 catalog_manager.cc:5484] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:33.887673 1577 raft_consensus.cc:482] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:33.887823 1577 raft_consensus.cc:504] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:33.888319 1577 leader_election.cc:288] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973), 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849)
I20240501 07:58:33.889029 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88887f8c213f46b2b4d4555d6a13c346" candidate_uuid: "10aab83192f447829c9b8254e36d94b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" is_pre_election: true
I20240501 07:58:33.889183 581 raft_consensus.cc:2454] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10aab83192f447829c9b8254e36d94b4 in term 0.
I20240501 07:58:33.889662 779 leader_election.cc:302] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10aab83192f447829c9b8254e36d94b4, 2c148dc68b5a4a38ba10be4e1e4b8cd2; no voters:
I20240501 07:58:33.889844 1577 raft_consensus.cc:2781] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:33.890031 1577 raft_consensus.cc:482] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:33.890086 1577 raft_consensus.cc:3037] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.890950 1577 raft_consensus.cc:504] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:33.891343 1577 leader_election.cc:288] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 1 election: Requested vote from peers e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973), 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849)
I20240501 07:58:33.891957 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88887f8c213f46b2b4d4555d6a13c346" candidate_uuid: "10aab83192f447829c9b8254e36d94b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2"
I20240501 07:58:33.892102 581 raft_consensus.cc:3037] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.892962 581 raft_consensus.cc:2454] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10aab83192f447829c9b8254e36d94b4 in term 1.
I20240501 07:58:33.892994 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88887f8c213f46b2b4d4555d6a13c346" candidate_uuid: "10aab83192f447829c9b8254e36d94b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5"
I20240501 07:58:33.892994 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88887f8c213f46b2b4d4555d6a13c346" candidate_uuid: "10aab83192f447829c9b8254e36d94b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5" is_pre_election: true
I20240501 07:58:33.893198 975 raft_consensus.cc:3037] T 88887f8c213f46b2b4d4555d6a13c346 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.893384 779 leader_election.cc:302] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10aab83192f447829c9b8254e36d94b4, 2c148dc68b5a4a38ba10be4e1e4b8cd2; no voters:
I20240501 07:58:33.893564 1577 raft_consensus.cc:2781] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:33.893754 1577 raft_consensus.cc:686] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 [term 1 LEADER]: Becoming Leader. State: Replica: 10aab83192f447829c9b8254e36d94b4, State: Running, Role: LEADER
I20240501 07:58:33.893904 1577 consensus_queue.cc:237] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:33.894202 975 raft_consensus.cc:2454] T 88887f8c213f46b2b4d4555d6a13c346 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10aab83192f447829c9b8254e36d94b4 in term 1.
I20240501 07:58:33.896136 439 catalog_manager.cc:5484] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10aab83192f447829c9b8254e36d94b4 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "10aab83192f447829c9b8254e36d94b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } health_report { overall_health: HEALTHY } } }
W20240501 07:58:34.069628 1754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:34.069924 1754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:34.073412 1754 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 07:58:34.130817 1720 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:34.136618 1711 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:34.163080 581 raft_consensus.cc:1261] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:34.163101 975 raft_consensus.cc:1261] T 5ab169c1536c4b849918c7cd593048c1 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:34.163636 1666 consensus_queue.cc:1035] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:34.163914 1725 consensus_queue.cc:1035] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:34.196440 581 raft_consensus.cc:1261] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Refusing update from remote peer 10aab83192f447829c9b8254e36d94b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:34.196967 1577 consensus_queue.cc:1035] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:34.199798 975 raft_consensus.cc:1261] T 88887f8c213f46b2b4d4555d6a13c346 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Refusing update from remote peer 10aab83192f447829c9b8254e36d94b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:34.200714 1577 consensus_queue.cc:1035] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 125.714 ms
time per row: 0.0628569 ms
I20240501 07:58:34.251554 439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35336:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:34.258702 561 tablet_service.cc:1467] Processing CreateTablet for tablet d6e4d59526a048fb8bb06d8e4a9de604 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0678a1a47ecc4cb3bb664b77ec709a59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.258925 823 tablet_service.cc:1467] Processing CreateTablet for tablet a25b951ae4804bc7bd1a8f1ef58e2bde (DEFAULT_TABLE table=rebalance_test_table_3 [id=0678a1a47ecc4cb3bb664b77ec709a59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.259064 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e4d59526a048fb8bb06d8e4a9de604. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.258925 824 tablet_service.cc:1467] Processing CreateTablet for tablet d6e4d59526a048fb8bb06d8e4a9de604 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0678a1a47ecc4cb3bb664b77ec709a59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.259234 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a25b951ae4804bc7bd1a8f1ef58e2bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.259444 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e4d59526a048fb8bb06d8e4a9de604. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.259816 825 tablet_service.cc:1467] Processing CreateTablet for tablet 004e01dbd0f94cbeb20cba08164d386d (DEFAULT_TABLE table=rebalance_test_table_3 [id=0678a1a47ecc4cb3bb664b77ec709a59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.260047 825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004e01dbd0f94cbeb20cba08164d386d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.260098 955 tablet_service.cc:1467] Processing CreateTablet for tablet a25b951ae4804bc7bd1a8f1ef58e2bde (DEFAULT_TABLE table=rebalance_test_table_3 [id=0678a1a47ecc4cb3bb664b77ec709a59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.260341 955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a25b951ae4804bc7bd1a8f1ef58e2bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.261698 954 tablet_service.cc:1467] Processing CreateTablet for tablet 004e01dbd0f94cbeb20cba08164d386d (DEFAULT_TABLE table=rebalance_test_table_3 [id=0678a1a47ecc4cb3bb664b77ec709a59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.261911 954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004e01dbd0f94cbeb20cba08164d386d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.262496 1814 tablet_bootstrap.cc:492] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5: Bootstrap starting.
I20240501 07:58:34.263027 692 tablet_service.cc:1467] Processing CreateTablet for tablet d6e4d59526a048fb8bb06d8e4a9de604 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0678a1a47ecc4cb3bb664b77ec709a59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.263283 692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e4d59526a048fb8bb06d8e4a9de604. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.263662 1814 tablet_bootstrap.cc:654] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.263926 1813 tablet_bootstrap.cc:492] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:34.264048 693 tablet_service.cc:1467] Processing CreateTablet for tablet 004e01dbd0f94cbeb20cba08164d386d (DEFAULT_TABLE table=rebalance_test_table_3 [id=0678a1a47ecc4cb3bb664b77ec709a59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.264292 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004e01dbd0f94cbeb20cba08164d386d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.264755 1813 tablet_bootstrap.cc:654] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.265374 1812 tablet_bootstrap.cc:492] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4: Bootstrap starting.
I20240501 07:58:34.266049 561 tablet_service.cc:1467] Processing CreateTablet for tablet a25b951ae4804bc7bd1a8f1ef58e2bde (DEFAULT_TABLE table=rebalance_test_table_3 [id=0678a1a47ecc4cb3bb664b77ec709a59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.266337 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a25b951ae4804bc7bd1a8f1ef58e2bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.267246 1813 tablet_bootstrap.cc:492] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: No bootstrap required, opened a new log
I20240501 07:58:34.267377 1813 ts_tablet_manager.cc:1367] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:58:34.267453 1814 tablet_bootstrap.cc:492] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5: No bootstrap required, opened a new log
I20240501 07:58:34.267545 1814 ts_tablet_manager.cc:1367] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:34.267884 1813 raft_consensus.cc:348] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:34.268002 1813 raft_consensus.cc:374] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.268044 1813 raft_consensus.cc:729] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.268034 1814 raft_consensus.cc:348] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.268131 1814 raft_consensus.cc:374] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.268164 1814 raft_consensus.cc:729] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2edc87d834f4e0bb662d501af69f2b5, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.268216 1813 consensus_queue.cc:260] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:34.268296 1814 consensus_queue.cc:260] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.268698 1814 ts_tablet_manager.cc:1398] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:34.268884 1814 tablet_bootstrap.cc:492] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5: Bootstrap starting.
I20240501 07:58:34.268965 1813 ts_tablet_manager.cc:1398] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:34.269160 1813 tablet_bootstrap.cc:492] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:34.270066 1817 tablet_bootstrap.cc:492] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02: Bootstrap starting.
I20240501 07:58:34.266407 1812 tablet_bootstrap.cc:654] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.271020 1817 tablet_bootstrap.cc:654] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.271173 1813 tablet_bootstrap.cc:654] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.271173 1814 tablet_bootstrap.cc:654] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.272337 1814 tablet_bootstrap.cc:492] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5: No bootstrap required, opened a new log
I20240501 07:58:34.272446 1814 ts_tablet_manager.cc:1367] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:34.272914 1814 raft_consensus.cc:348] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:34.272989 1813 tablet_bootstrap.cc:492] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2: No bootstrap required, opened a new log
I20240501 07:58:34.273041 1814 raft_consensus.cc:374] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.273082 1813 ts_tablet_manager.cc:1367] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 07:58:34.273080 1814 raft_consensus.cc:729] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2edc87d834f4e0bb662d501af69f2b5, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.273197 1814 consensus_queue.cc:260] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:34.273491 1814 ts_tablet_manager.cc:1398] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:34.273504 1813 raft_consensus.cc:348] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.273602 1813 raft_consensus.cc:374] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.273633 1813 raft_consensus.cc:729] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.273870 1813 consensus_queue.cc:260] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.274387 1812 tablet_bootstrap.cc:492] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4: No bootstrap required, opened a new log
I20240501 07:58:34.274500 1813 ts_tablet_manager.cc:1398] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:34.274530 1812 ts_tablet_manager.cc:1367] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:58:34.275282 1817 tablet_bootstrap.cc:492] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02: No bootstrap required, opened a new log
I20240501 07:58:34.275393 1817 ts_tablet_manager.cc:1367] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:34.275655 1812 raft_consensus.cc:348] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:34.275769 1812 raft_consensus.cc:374] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.275800 1812 raft_consensus.cc:729] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10aab83192f447829c9b8254e36d94b4, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.275781 1817 raft_consensus.cc:348] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:34.275892 1817 raft_consensus.cc:374] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.275923 1817 raft_consensus.cc:729] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.275964 1812 consensus_queue.cc:260] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:34.276240 1817 consensus_queue.cc:260] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:34.276602 1817 ts_tablet_manager.cc:1398] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:34.276938 1817 tablet_bootstrap.cc:492] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02: Bootstrap starting.
I20240501 07:58:34.277627 1812 ts_tablet_manager.cc:1398] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:34.277814 1812 tablet_bootstrap.cc:492] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4: Bootstrap starting.
I20240501 07:58:34.278024 1817 tablet_bootstrap.cc:654] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.279117 1817 tablet_bootstrap.cc:492] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02: No bootstrap required, opened a new log
I20240501 07:58:34.279206 1817 ts_tablet_manager.cc:1367] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:34.279644 1817 raft_consensus.cc:348] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:34.279757 1817 raft_consensus.cc:374] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.279798 1817 raft_consensus.cc:729] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.279937 1817 consensus_queue.cc:260] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:34.280263 1817 ts_tablet_manager.cc:1398] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:34.280350 1812 tablet_bootstrap.cc:654] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.281562 1812 tablet_bootstrap.cc:492] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4: No bootstrap required, opened a new log
I20240501 07:58:34.281801 1812 ts_tablet_manager.cc:1367] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 07:58:34.282279 1812 raft_consensus.cc:348] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:34.282392 1812 raft_consensus.cc:374] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.282471 1812 raft_consensus.cc:729] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10aab83192f447829c9b8254e36d94b4, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.282603 1812 consensus_queue.cc:260] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:34.282986 1812 ts_tablet_manager.cc:1398] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:34.283136 1812 tablet_bootstrap.cc:492] T a25b951ae4804bc7bd1a8f1ef58e2bde P 10aab83192f447829c9b8254e36d94b4: Bootstrap starting.
I20240501 07:58:34.284304 1812 tablet_bootstrap.cc:654] T a25b951ae4804bc7bd1a8f1ef58e2bde P 10aab83192f447829c9b8254e36d94b4: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.285945 1812 tablet_bootstrap.cc:492] T a25b951ae4804bc7bd1a8f1ef58e2bde P 10aab83192f447829c9b8254e36d94b4: No bootstrap required, opened a new log
I20240501 07:58:34.286036 1812 ts_tablet_manager.cc:1367] T a25b951ae4804bc7bd1a8f1ef58e2bde P 10aab83192f447829c9b8254e36d94b4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:34.286396 1812 raft_consensus.cc:348] T a25b951ae4804bc7bd1a8f1ef58e2bde P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.286540 1812 raft_consensus.cc:374] T a25b951ae4804bc7bd1a8f1ef58e2bde P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.286563 1812 raft_consensus.cc:729] T a25b951ae4804bc7bd1a8f1ef58e2bde P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10aab83192f447829c9b8254e36d94b4, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.286679 1812 consensus_queue.cc:260] T a25b951ae4804bc7bd1a8f1ef58e2bde P 10aab83192f447829c9b8254e36d94b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.286949 1812 ts_tablet_manager.cc:1398] T a25b951ae4804bc7bd1a8f1ef58e2bde P 10aab83192f447829c9b8254e36d94b4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:34.294344 1666 raft_consensus.cc:482] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:34.294548 1666 raft_consensus.cc:504] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:34.294978 1666 leader_election.cc:288] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475), e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973)
I20240501 07:58:34.295509 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004e01dbd0f94cbeb20cba08164d386d" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4" is_pre_election: true
I20240501 07:58:34.295509 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004e01dbd0f94cbeb20cba08164d386d" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5" is_pre_election: true
I20240501 07:58:34.295661 843 raft_consensus.cc:2454] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 0.
I20240501 07:58:34.295661 976 raft_consensus.cc:2454] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 0.
I20240501 07:58:34.296006 649 leader_election.cc:302] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2c804beb8ef40ecb16c952f8875ca02, e2edc87d834f4e0bb662d501af69f2b5; no voters:
I20240501 07:58:34.296185 1666 raft_consensus.cc:2781] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:34.296254 1666 raft_consensus.cc:482] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:34.296280 1666 raft_consensus.cc:3037] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.297116 1666 raft_consensus.cc:504] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:34.297497 1666 leader_election.cc:288] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 election: Requested vote from peers 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475), e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973)
I20240501 07:58:34.298000 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004e01dbd0f94cbeb20cba08164d386d" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5"
I20240501 07:58:34.297987 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004e01dbd0f94cbeb20cba08164d386d" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4"
I20240501 07:58:34.298130 843 raft_consensus.cc:3037] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.298131 976 raft_consensus.cc:3037] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.299211 976 raft_consensus.cc:2454] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 1.
I20240501 07:58:34.299211 843 raft_consensus.cc:2454] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 1.
I20240501 07:58:34.299775 647 leader_election.cc:302] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10aab83192f447829c9b8254e36d94b4, e2c804beb8ef40ecb16c952f8875ca02; no voters:
I20240501 07:58:34.300251 1666 raft_consensus.cc:2781] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:34.300343 1666 raft_consensus.cc:686] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Becoming Leader. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Running, Role: LEADER
I20240501 07:58:34.300535 1666 consensus_queue.cc:237] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:34.302940 437 catalog_manager.cc:5484] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:34.453352 1798 raft_consensus.cc:482] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:34.453554 1798 raft_consensus.cc:504] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:34.454380 1798 leader_election.cc:288] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475), e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577)
I20240501 07:58:34.454936 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e4d59526a048fb8bb06d8e4a9de604" candidate_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02" is_pre_election: true
I20240501 07:58:34.455108 713 raft_consensus.cc:2454] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c148dc68b5a4a38ba10be4e1e4b8cd2 in term 0.
I20240501 07:58:34.455410 517 leader_election.cc:302] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2, e2c804beb8ef40ecb16c952f8875ca02; no voters:
I20240501 07:58:34.455607 1798 raft_consensus.cc:2781] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:34.455660 1798 raft_consensus.cc:482] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:34.455698 1798 raft_consensus.cc:3037] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.456557 1798 raft_consensus.cc:504] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:34.456952 1798 leader_election.cc:288] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [CANDIDATE]: Term 1 election: Requested vote from peers 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475), e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577)
I20240501 07:58:34.457315 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e4d59526a048fb8bb06d8e4a9de604" candidate_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02"
I20240501 07:58:34.457440 713 raft_consensus.cc:3037] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.458307 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e4d59526a048fb8bb06d8e4a9de604" candidate_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4" is_pre_election: true
I20240501 07:58:34.458314 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e4d59526a048fb8bb06d8e4a9de604" candidate_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4"
I20240501 07:58:34.458509 843 raft_consensus.cc:2454] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c148dc68b5a4a38ba10be4e1e4b8cd2 in term 0.
I20240501 07:58:34.458590 713 raft_consensus.cc:2454] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c148dc68b5a4a38ba10be4e1e4b8cd2 in term 1.
I20240501 07:58:34.458925 517 leader_election.cc:302] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2, e2c804beb8ef40ecb16c952f8875ca02; no voters:
I20240501 07:58:34.459177 1798 raft_consensus.cc:2781] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:34.459290 1798 raft_consensus.cc:686] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 LEADER]: Becoming Leader. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Running, Role: LEADER
I20240501 07:58:34.459463 1798 consensus_queue.cc:237] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:34.461354 437 catalog_manager.cc:5484] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:34.502393 1782 raft_consensus.cc:482] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:34.502576 1782 raft_consensus.cc:504] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.503101 1782 leader_election.cc:288] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475), 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849)
I20240501 07:58:34.503594 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25b951ae4804bc7bd1a8f1ef58e2bde" candidate_uuid: "e2edc87d834f4e0bb662d501af69f2b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" is_pre_election: true
I20240501 07:58:34.503620 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25b951ae4804bc7bd1a8f1ef58e2bde" candidate_uuid: "e2edc87d834f4e0bb662d501af69f2b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4" is_pre_election: true
I20240501 07:58:34.503739 582 raft_consensus.cc:2454] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2edc87d834f4e0bb662d501af69f2b5 in term 0.
I20240501 07:58:34.503787 845 raft_consensus.cc:2454] T a25b951ae4804bc7bd1a8f1ef58e2bde P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2edc87d834f4e0bb662d501af69f2b5 in term 0.
I20240501 07:58:34.504091 909 leader_election.cc:302] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10aab83192f447829c9b8254e36d94b4, e2edc87d834f4e0bb662d501af69f2b5; no voters:
I20240501 07:58:34.504356 1782 raft_consensus.cc:2781] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:34.504449 1782 raft_consensus.cc:482] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:34.504508 1782 raft_consensus.cc:3037] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.505437 1782 raft_consensus.cc:504] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.505810 1782 leader_election.cc:288] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 1 election: Requested vote from peers 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475), 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849)
I20240501 07:58:34.506683 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25b951ae4804bc7bd1a8f1ef58e2bde" candidate_uuid: "e2edc87d834f4e0bb662d501af69f2b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2"
I20240501 07:58:34.506845 582 raft_consensus.cc:3037] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.506791 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a25b951ae4804bc7bd1a8f1ef58e2bde" candidate_uuid: "e2edc87d834f4e0bb662d501af69f2b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4"
I20240501 07:58:34.506942 845 raft_consensus.cc:3037] T a25b951ae4804bc7bd1a8f1ef58e2bde P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.507670 845 raft_consensus.cc:2454] T a25b951ae4804bc7bd1a8f1ef58e2bde P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2edc87d834f4e0bb662d501af69f2b5 in term 1.
I20240501 07:58:34.507680 582 raft_consensus.cc:2454] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2edc87d834f4e0bb662d501af69f2b5 in term 1.
I20240501 07:58:34.507966 910 leader_election.cc:302] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2, e2edc87d834f4e0bb662d501af69f2b5; no voters:
I20240501 07:58:34.508162 1782 raft_consensus.cc:2781] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:34.508261 1782 raft_consensus.cc:686] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 [term 1 LEADER]: Becoming Leader. State: Replica: e2edc87d834f4e0bb662d501af69f2b5, State: Running, Role: LEADER
I20240501 07:58:34.508443 1782 consensus_queue.cc:237] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.510259 440 catalog_manager.cc:5484] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "e2edc87d834f4e0bb662d501af69f2b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:34.697854 1829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:34.698172 1829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:34.701658 1829 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 07:58:34.790810 1666 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:34.794780 1725 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:34.809578 582 raft_consensus.cc:1261] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:34.810108 1718 consensus_queue.cc:1035] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:34.811203 845 raft_consensus.cc:1261] T a25b951ae4804bc7bd1a8f1ef58e2bde P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:34.813330 713 raft_consensus.cc:1261] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Refusing update from remote peer 2c148dc68b5a4a38ba10be4e1e4b8cd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:34.813777 1798 consensus_queue.cc:1035] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:34.815315 1718 consensus_queue.cc:1035] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:34.816356 845 raft_consensus.cc:3037] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.817431 845 raft_consensus.cc:1261] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Refusing update from remote peer 2c148dc68b5a4a38ba10be4e1e4b8cd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:34.819237 1798 consensus_queue.cc:1035] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 118.241 ms
time per row: 0.0591203 ms
I20240501 07:58:34.877774 437 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35336:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:34.884346 693 tablet_service.cc:1467] Processing CreateTablet for tablet 4f220c7dad9d4cf390608b13c5de0fbf (DEFAULT_TABLE table=rebalance_test_table_4 [id=e4d5b126c14f4ca28bb69545ef633b2a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.884346 823 tablet_service.cc:1467] Processing CreateTablet for tablet 4f220c7dad9d4cf390608b13c5de0fbf (DEFAULT_TABLE table=rebalance_test_table_4 [id=e4d5b126c14f4ca28bb69545ef633b2a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.884343 561 tablet_service.cc:1467] Processing CreateTablet for tablet 4f220c7dad9d4cf390608b13c5de0fbf (DEFAULT_TABLE table=rebalance_test_table_4 [id=e4d5b126c14f4ca28bb69545ef633b2a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.884692 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f220c7dad9d4cf390608b13c5de0fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.884692 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f220c7dad9d4cf390608b13c5de0fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.884696 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f220c7dad9d4cf390608b13c5de0fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.884969 825 tablet_service.cc:1467] Processing CreateTablet for tablet 46ad498fe543450f8911b83dd71d07f3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e4d5b126c14f4ca28bb69545ef633b2a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.885169 825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ad498fe543450f8911b83dd71d07f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.886731 954 tablet_service.cc:1467] Processing CreateTablet for tablet 789ccd2b398d450884458b02e51039e7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e4d5b126c14f4ca28bb69545ef633b2a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.886807 692 tablet_service.cc:1467] Processing CreateTablet for tablet 789ccd2b398d450884458b02e51039e7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e4d5b126c14f4ca28bb69545ef633b2a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.887020 954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789ccd2b398d450884458b02e51039e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.887073 692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789ccd2b398d450884458b02e51039e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.889461 563 tablet_service.cc:1467] Processing CreateTablet for tablet 46ad498fe543450f8911b83dd71d07f3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e4d5b126c14f4ca28bb69545ef633b2a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.889461 562 tablet_service.cc:1467] Processing CreateTablet for tablet 789ccd2b398d450884458b02e51039e7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e4d5b126c14f4ca28bb69545ef633b2a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.889653 1900 tablet_bootstrap.cc:492] T 789ccd2b398d450884458b02e51039e7 P e2edc87d834f4e0bb662d501af69f2b5: Bootstrap starting.
I20240501 07:58:34.889807 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ad498fe543450f8911b83dd71d07f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.889923 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789ccd2b398d450884458b02e51039e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.890659 1900 tablet_bootstrap.cc:654] T 789ccd2b398d450884458b02e51039e7 P e2edc87d834f4e0bb662d501af69f2b5: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.886731 955 tablet_service.cc:1467] Processing CreateTablet for tablet 46ad498fe543450f8911b83dd71d07f3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e4d5b126c14f4ca28bb69545ef633b2a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.891099 955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ad498fe543450f8911b83dd71d07f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.892102 1900 tablet_bootstrap.cc:492] T 789ccd2b398d450884458b02e51039e7 P e2edc87d834f4e0bb662d501af69f2b5: No bootstrap required, opened a new log
I20240501 07:58:34.892247 1900 ts_tablet_manager.cc:1367] T 789ccd2b398d450884458b02e51039e7 P e2edc87d834f4e0bb662d501af69f2b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:34.893046 1900 raft_consensus.cc:348] T 789ccd2b398d450884458b02e51039e7 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:34.893188 1900 raft_consensus.cc:374] T 789ccd2b398d450884458b02e51039e7 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.893221 1900 raft_consensus.cc:729] T 789ccd2b398d450884458b02e51039e7 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2edc87d834f4e0bb662d501af69f2b5, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.893422 1900 consensus_queue.cc:260] T 789ccd2b398d450884458b02e51039e7 P e2edc87d834f4e0bb662d501af69f2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:34.893816 1900 ts_tablet_manager.cc:1398] T 789ccd2b398d450884458b02e51039e7 P e2edc87d834f4e0bb662d501af69f2b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:34.893988 1900 tablet_bootstrap.cc:492] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5: Bootstrap starting.
I20240501 07:58:34.895149 1903 tablet_bootstrap.cc:492] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:34.895467 1900 tablet_bootstrap.cc:654] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.894601 1899 tablet_bootstrap.cc:492] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02: Bootstrap starting.
I20240501 07:58:34.896241 1903 tablet_bootstrap.cc:654] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.896663 1900 tablet_bootstrap.cc:492] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5: No bootstrap required, opened a new log
I20240501 07:58:34.896760 1900 ts_tablet_manager.cc:1367] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:34.896876 1899 tablet_bootstrap.cc:654] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.897270 1900 raft_consensus.cc:348] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.897395 1900 raft_consensus.cc:374] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.897449 1900 raft_consensus.cc:729] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2edc87d834f4e0bb662d501af69f2b5, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.897372 1903 tablet_bootstrap.cc:492] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2: No bootstrap required, opened a new log
I20240501 07:58:34.897578 1901 tablet_bootstrap.cc:492] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4: Bootstrap starting.
I20240501 07:58:34.897584 1900 consensus_queue.cc:260] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.897661 1903 ts_tablet_manager.cc:1367] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:34.897845 1900 ts_tablet_manager.cc:1398] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:34.898138 1903 raft_consensus.cc:348] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.898248 1903 raft_consensus.cc:374] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.898293 1903 raft_consensus.cc:729] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.898638 1903 consensus_queue.cc:260] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.898866 1899 tablet_bootstrap.cc:492] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02: No bootstrap required, opened a new log
I20240501 07:58:34.898967 1899 ts_tablet_manager.cc:1367] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:34.899036 1901 tablet_bootstrap.cc:654] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.899415 1899 raft_consensus.cc:348] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.899597 1899 raft_consensus.cc:374] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.899652 1899 raft_consensus.cc:729] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.899821 1899 consensus_queue.cc:260] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.900151 1899 ts_tablet_manager.cc:1398] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:34.900153 1901 tablet_bootstrap.cc:492] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4: No bootstrap required, opened a new log
I20240501 07:58:34.900250 1901 ts_tablet_manager.cc:1367] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:34.900389 1899 tablet_bootstrap.cc:492] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02: Bootstrap starting.
I20240501 07:58:34.900697 1901 raft_consensus.cc:348] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.900766 1903 ts_tablet_manager.cc:1398] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:34.900805 1901 raft_consensus.cc:374] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.900839 1901 raft_consensus.cc:729] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10aab83192f447829c9b8254e36d94b4, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.900969 1901 consensus_queue.cc:260] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.901223 1901 ts_tablet_manager.cc:1398] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:34.901345 1899 tablet_bootstrap.cc:654] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.901397 1901 tablet_bootstrap.cc:492] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4: Bootstrap starting.
I20240501 07:58:34.902266 1901 tablet_bootstrap.cc:654] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.903998 1903 tablet_bootstrap.cc:492] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:34.904204 1901 tablet_bootstrap.cc:492] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4: No bootstrap required, opened a new log
I20240501 07:58:34.904302 1901 ts_tablet_manager.cc:1367] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:34.904569 1899 tablet_bootstrap.cc:492] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02: No bootstrap required, opened a new log
I20240501 07:58:34.904727 1899 ts_tablet_manager.cc:1367] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:34.904836 1901 raft_consensus.cc:348] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.904958 1901 raft_consensus.cc:374] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.904997 1901 raft_consensus.cc:729] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10aab83192f447829c9b8254e36d94b4, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.905100 1903 tablet_bootstrap.cc:654] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.905122 1901 consensus_queue.cc:260] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.905228 1899 raft_consensus.cc:348] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:34.905351 1899 raft_consensus.cc:374] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.905411 1901 ts_tablet_manager.cc:1398] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:34.905417 1899 raft_consensus.cc:729] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.905596 1899 consensus_queue.cc:260] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:34.906002 1899 ts_tablet_manager.cc:1398] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:34.908177 1903 tablet_bootstrap.cc:492] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: No bootstrap required, opened a new log
I20240501 07:58:34.908285 1903 ts_tablet_manager.cc:1367] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:34.908890 1903 raft_consensus.cc:348] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.909006 1903 raft_consensus.cc:374] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.909034 1903 raft_consensus.cc:729] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.909451 1903 consensus_queue.cc:260] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:34.909763 1903 ts_tablet_manager.cc:1398] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:34.909991 1903 tablet_bootstrap.cc:492] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:34.911783 1903 tablet_bootstrap.cc:654] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.913789 1903 tablet_bootstrap.cc:492] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: No bootstrap required, opened a new log
I20240501 07:58:34.913928 1903 ts_tablet_manager.cc:1367] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 07:58:34.914388 1903 raft_consensus.cc:348] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:34.914549 1903 raft_consensus.cc:374] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.914583 1903 raft_consensus.cc:729] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.914728 1903 consensus_queue.cc:260] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:34.914992 1903 ts_tablet_manager.cc:1398] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:34.922169 1883 raft_consensus.cc:482] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:34.922545 1883 raft_consensus.cc:504] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:34.922999 1883 leader_election.cc:288] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849), e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973)
I20240501 07:58:34.923458 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789ccd2b398d450884458b02e51039e7" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5" is_pre_election: true
I20240501 07:58:34.923457 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789ccd2b398d450884458b02e51039e7" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" is_pre_election: true
I20240501 07:58:34.923596 976 raft_consensus.cc:2454] T 789ccd2b398d450884458b02e51039e7 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 0.
I20240501 07:58:34.923609 582 raft_consensus.cc:2454] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 0.
I20240501 07:58:34.923935 648 leader_election.cc:302] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2, e2c804beb8ef40ecb16c952f8875ca02; no voters:
I20240501 07:58:34.924108 1883 raft_consensus.cc:2781] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:34.924171 1883 raft_consensus.cc:482] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:34.924208 1883 raft_consensus.cc:3037] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.925042 1883 raft_consensus.cc:504] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:34.925433 1883 leader_election.cc:288] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 election: Requested vote from peers 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849), e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973)
I20240501 07:58:34.925951 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789ccd2b398d450884458b02e51039e7" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5"
I20240501 07:58:34.926095 976 raft_consensus.cc:3037] T 789ccd2b398d450884458b02e51039e7 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.926118 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789ccd2b398d450884458b02e51039e7" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2"
I20240501 07:58:34.926249 582 raft_consensus.cc:3037] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.927095 976 raft_consensus.cc:2454] T 789ccd2b398d450884458b02e51039e7 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 1.
I20240501 07:58:34.927421 582 raft_consensus.cc:2454] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 1.
I20240501 07:58:34.927433 649 leader_election.cc:302] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2c804beb8ef40ecb16c952f8875ca02, e2edc87d834f4e0bb662d501af69f2b5; no voters:
I20240501 07:58:34.927604 1883 raft_consensus.cc:2781] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:34.927695 1883 raft_consensus.cc:686] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Becoming Leader. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Running, Role: LEADER
I20240501 07:58:34.927835 1883 consensus_queue.cc:237] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:34.929672 440 catalog_manager.cc:5484] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:35.032737 1711 raft_consensus.cc:482] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:35.032908 1711 raft_consensus.cc:504] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:35.033445 1711 leader_election.cc:288] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577), 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475)
I20240501 07:58:35.033990 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f220c7dad9d4cf390608b13c5de0fbf" candidate_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02" is_pre_election: true
I20240501 07:58:35.034193 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f220c7dad9d4cf390608b13c5de0fbf" candidate_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4" is_pre_election: true
I20240501 07:58:35.034329 713 raft_consensus.cc:2454] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c148dc68b5a4a38ba10be4e1e4b8cd2 in term 0.
I20240501 07:58:35.034350 845 raft_consensus.cc:2454] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c148dc68b5a4a38ba10be4e1e4b8cd2 in term 0.
I20240501 07:58:35.034840 517 leader_election.cc:302] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10aab83192f447829c9b8254e36d94b4, 2c148dc68b5a4a38ba10be4e1e4b8cd2, e2c804beb8ef40ecb16c952f8875ca02; no voters:
I20240501 07:58:35.035132 1711 raft_consensus.cc:2781] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:35.035214 1711 raft_consensus.cc:482] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:35.035241 1711 raft_consensus.cc:3037] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.036142 1711 raft_consensus.cc:504] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:35.036586 1711 leader_election.cc:288] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [CANDIDATE]: Term 1 election: Requested vote from peers e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577), 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475)
I20240501 07:58:35.037125 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f220c7dad9d4cf390608b13c5de0fbf" candidate_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4"
I20240501 07:58:35.037123 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f220c7dad9d4cf390608b13c5de0fbf" candidate_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02"
I20240501 07:58:35.037305 845 raft_consensus.cc:3037] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.037305 713 raft_consensus.cc:3037] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.038286 845 raft_consensus.cc:2454] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c148dc68b5a4a38ba10be4e1e4b8cd2 in term 1.
I20240501 07:58:35.038286 713 raft_consensus.cc:2454] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c148dc68b5a4a38ba10be4e1e4b8cd2 in term 1.
I20240501 07:58:35.038882 516 leader_election.cc:302] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10aab83192f447829c9b8254e36d94b4, 2c148dc68b5a4a38ba10be4e1e4b8cd2, e2c804beb8ef40ecb16c952f8875ca02; no voters:
I20240501 07:58:35.039141 1711 raft_consensus.cc:2781] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:35.039261 1711 raft_consensus.cc:686] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 LEADER]: Becoming Leader. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Running, Role: LEADER
I20240501 07:58:35.039434 1711 consensus_queue.cc:237] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:35.041512 440 catalog_manager.cc:5484] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:35.059196 1711 raft_consensus.cc:482] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:35.059406 1711 raft_consensus.cc:504] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:35.059984 1711 leader_election.cc:288] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475), e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973)
I20240501 07:58:35.060700 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ad498fe543450f8911b83dd71d07f3" candidate_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5" is_pre_election: true
I20240501 07:58:35.060720 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ad498fe543450f8911b83dd71d07f3" candidate_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4" is_pre_election: true
I20240501 07:58:35.060873 976 raft_consensus.cc:2454] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c148dc68b5a4a38ba10be4e1e4b8cd2 in term 0.
I20240501 07:58:35.060945 845 raft_consensus.cc:2454] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c148dc68b5a4a38ba10be4e1e4b8cd2 in term 0.
I20240501 07:58:35.061241 518 leader_election.cc:302] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2, e2edc87d834f4e0bb662d501af69f2b5; no voters:
I20240501 07:58:35.061507 1711 raft_consensus.cc:2781] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:35.061558 1711 raft_consensus.cc:482] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:35.061591 1711 raft_consensus.cc:3037] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.062748 1711 raft_consensus.cc:504] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:35.063175 1711 leader_election.cc:288] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [CANDIDATE]: Term 1 election: Requested vote from peers 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475), e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973)
I20240501 07:58:35.063690 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ad498fe543450f8911b83dd71d07f3" candidate_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4"
I20240501 07:58:35.063887 845 raft_consensus.cc:3037] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.064028 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ad498fe543450f8911b83dd71d07f3" candidate_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5"
I20240501 07:58:35.064157 976 raft_consensus.cc:3037] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:35.064972 845 raft_consensus.cc:2454] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c148dc68b5a4a38ba10be4e1e4b8cd2 in term 1.
I20240501 07:58:35.064992 976 raft_consensus.cc:2454] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c148dc68b5a4a38ba10be4e1e4b8cd2 in term 1.
I20240501 07:58:35.065402 518 leader_election.cc:302] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2, e2edc87d834f4e0bb662d501af69f2b5; no voters:
I20240501 07:58:35.065699 1711 raft_consensus.cc:2781] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:35.065862 1711 raft_consensus.cc:686] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 LEADER]: Becoming Leader. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Running, Role: LEADER
I20240501 07:58:35.066056 1711 consensus_queue.cc:237] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:35.068271 440 catalog_manager.cc:5484] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } health_report { overall_health: HEALTHY } } }
W20240501 07:58:35.211481 1915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:35.211848 1915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:35.215597 1915 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 07:58:35.303218 976 raft_consensus.cc:1261] T 789ccd2b398d450884458b02e51039e7 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:35.303459 845 raft_consensus.cc:1261] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Refusing update from remote peer 2c148dc68b5a4a38ba10be4e1e4b8cd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:35.303966 1798 consensus_queue.cc:1035] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:35.304338 1882 consensus_queue.cc:1035] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:35.304749 713 raft_consensus.cc:1261] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Refusing update from remote peer 2c148dc68b5a4a38ba10be4e1e4b8cd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:35.304844 582 raft_consensus.cc:1261] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:35.305243 1798 consensus_queue.cc:1035] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:35.306865 976 raft_consensus.cc:1261] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Refusing update from remote peer 2c148dc68b5a4a38ba10be4e1e4b8cd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:35.308537 1798 consensus_queue.cc:1035] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:35.309134 843 raft_consensus.cc:1261] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Refusing update from remote peer 2c148dc68b5a4a38ba10be4e1e4b8cd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:35.309116 1873 consensus_queue.cc:1035] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:35.310087 1798 consensus_queue.cc:1035] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:35.339212 1871 mvcc.cc:205] Tried to move back new op lower bound from 7022798091494576128 to 7022798089945567232. Current Snapshot: MvccSnapshot[applied={T|T < 7022798091476713472}]
INSERT report
rows total: 2000
time total: 114.139 ms
time per row: 0.0570696 ms
I20240501 07:58:35.382161 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:39953
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:35609
--webserver_port=33613
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 07:58:35.505981 1949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:35.506228 1949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:35.506296 1949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:35.509826 1949 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:35.509917 1949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:35.510022 1949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:58:35.514221 1949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:39953
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=33613
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35609
--never_fsync=true
--heap_profile_path=/tmp/kudu.1949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:35.515286 1949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:35.516517 1949 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 07:58:35.526835 1949 server_base.cc:1034] running on GCE node
W20240501 07:58:35.527251 1954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:35.527966 1957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:35.528424 1955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:35.528946 1949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:35.529709 1949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:35.530889 1949 hybrid_clock.cc:648] HybridClock initialized: now 1714550315530861 us; error 54 us; skew 500 ppm
I20240501 07:58:35.533097 1949 webserver.cc:460] Webserver started at http://127.0.99.133:33613/ using document root <none> and password file <none>
I20240501 07:58:35.533758 1949 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:35.533836 1949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:35.537914 1949 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:35.540258 1963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:35.541631 1949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:35.541774 1949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b331ee755f7d4df48eb9f4460a0340b6"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-2srn"
I20240501 07:58:35.542277 1949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:35.564834 1949 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:35.565616 1949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:35.565811 1949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:35.566499 1949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:35.567703 1949 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:35.567762 1949 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:35.567816 1949 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:35.567842 1949 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:35.578353 1949 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:39953
I20240501 07:58:35.578465 2076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:39953 every 8 connection(s)
I20240501 07:58:35.580536 1949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:58:35.589502 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 1949
I20240501 07:58:35.589996 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:39945
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:35609
--webserver_port=45131
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 07:58:35.592500 2077 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35609
I20240501 07:58:35.592782 2077 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:35.593422 2077 heartbeater.cc:511] Master 127.0.99.190:35609 requested a full tablet report, sending...
I20240501 07:58:35.595363 440 ts_manager.cc:194] Re-registered known tserver with Master: b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133:39953)
I20240501 07:58:35.596207 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:43119
W20240501 07:58:35.715679 2080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:35.715977 2080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:35.716049 2080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:35.720165 2080 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:35.720276 2080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:35.720384 2080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 07:58:35.724746 2080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:39945
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=45131
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35609
--never_fsync=true
--heap_profile_path=/tmp/kudu.2080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:35.725889 2080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:35.727238 2080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:35.737983 2085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:35.738328 2086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:35.738876 2088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:35.739490 2080 server_base.cc:1034] running on GCE node
I20240501 07:58:35.740082 2080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:35.740856 2080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:35.742065 2080 hybrid_clock.cc:648] HybridClock initialized: now 1714550315742030 us; error 61 us; skew 500 ppm
I20240501 07:58:35.744962 2080 webserver.cc:460] Webserver started at http://127.0.99.134:45131/ using document root <none> and password file <none>
I20240501 07:58:35.745836 2080 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:35.745940 2080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:35.750157 2080 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 07:58:35.752570 2094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:35.753773 2080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:35.753901 2080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "36272be7c71b413a94d36b4257d34425"
format_stamp: "Formatted at 2024-05-01 07:58:28 on dist-test-slave-2srn"
I20240501 07:58:35.754364 2080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:35.777070 2080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:35.777827 2080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:35.778003 2080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:35.778702 2080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:35.779783 2080 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:35.779840 2080 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:35.779891 2080 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:35.779920 2080 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:35.790223 2080 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:39945
I20240501 07:58:35.790331 2207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:39945 every 8 connection(s)
I20240501 07:58:35.791533 2080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 07:58:35.798458 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 2080
I20240501 07:58:35.798910 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:44501
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:35609
--webserver_port=40893
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 07:58:35.803505 2208 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35609
I20240501 07:58:35.803815 2208 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:35.804563 2208 heartbeater.cc:511] Master 127.0.99.190:35609 requested a full tablet report, sending...
I20240501 07:58:35.805848 440 ts_manager.cc:194] Re-registered known tserver with Master: 36272be7c71b413a94d36b4257d34425 (127.0.99.134:39945)
I20240501 07:58:35.806695 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:45559
W20240501 07:58:35.924865 2211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:35.925109 2211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:35.925179 2211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:35.928774 2211 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:35.928869 2211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:35.928970 2211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 07:58:35.933012 2211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:44501
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=40893
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35609
--never_fsync=true
--heap_profile_path=/tmp/kudu.2211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:35.934099 2211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:35.935444 2211 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:35.945214 2217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:35.945331 2216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:35.945799 2219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:35.946725 2211 server_base.cc:1034] running on GCE node
I20240501 07:58:35.947417 2211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:35.948519 2211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:35.949759 2211 hybrid_clock.cc:648] HybridClock initialized: now 1714550315949735 us; error 52 us; skew 500 ppm
I20240501 07:58:35.952271 2211 webserver.cc:460] Webserver started at http://127.0.99.135:40893/ using document root <none> and password file <none>
I20240501 07:58:35.952872 2211 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:35.952957 2211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:35.956722 2211 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20240501 07:58:35.958957 2225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:35.960289 2211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 07:58:35.960422 2211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "c1b68243a7144af5835b051aa9916d13"
format_stamp: "Formatted at 2024-05-01 07:58:28 on dist-test-slave-2srn"
I20240501 07:58:35.960901 2211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:35.983600 2211 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:35.984233 2211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:35.984380 2211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:35.984920 2211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:35.985875 2211 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:35.985925 2211 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:35.985980 2211 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:35.986011 2211 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:35.994763 2211 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:44501
I20240501 07:58:35.994779 2338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:44501 every 8 connection(s)
I20240501 07:58:35.995858 2211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 07:58:35.996167 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 2211
I20240501 07:58:35.996609 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:44643
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:35609
--webserver_port=34637
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 07:58:36.006942 2339 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35609
I20240501 07:58:36.007241 2339 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:36.007994 2339 heartbeater.cc:511] Master 127.0.99.190:35609 requested a full tablet report, sending...
I20240501 07:58:36.009249 440 ts_manager.cc:194] Re-registered known tserver with Master: c1b68243a7144af5835b051aa9916d13 (127.0.99.135:44501)
I20240501 07:58:36.009825 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:44957
W20240501 07:58:36.117691 2341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:36.117936 2341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:36.118000 2341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:36.121330 2341 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:36.121431 2341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:36.121507 2341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240501 07:58:36.125715 2341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:44643
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=34637
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35609
--never_fsync=true
--heap_profile_path=/tmp/kudu.2341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:36.126854 2341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:36.128029 2341 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:36.137220 2348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:36.137620 2347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:36.138047 2350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:36.138653 2341 server_base.cc:1034] running on GCE node
I20240501 07:58:36.139165 2341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:36.139956 2341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:36.141175 2341 hybrid_clock.cc:648] HybridClock initialized: now 1714550316141151 us; error 36 us; skew 500 ppm
I20240501 07:58:36.143249 2341 webserver.cc:460] Webserver started at http://127.0.99.136:34637/ using document root <none> and password file <none>
I20240501 07:58:36.143853 2341 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:36.143926 2341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:36.147608 2341 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 07:58:36.150012 2356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:36.151207 2341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:36.151329 2341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "6645111fc9e446e8877750a3c4779750"
format_stamp: "Formatted at 2024-05-01 07:58:28 on dist-test-slave-2srn"
I20240501 07:58:36.151795 2341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:36.172544 2341 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:36.173173 2341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:36.173327 2341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:36.173867 2341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:36.174832 2341 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:36.174881 2341 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:36.174935 2341 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:36.174981 2341 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:36.183629 2341 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:44643
I20240501 07:58:36.183650 2469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:44643 every 8 connection(s)
I20240501 07:58:36.184717 2341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 07:58:36.193640 2470 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35609
I20240501 07:58:36.193914 2470 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:36.194509 2470 heartbeater.cc:511] Master 127.0.99.190:35609 requested a full tablet report, sending...
I20240501 07:58:36.194533 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 2341
I20240501 07:58:36.195487 440 ts_manager.cc:194] Re-registered known tserver with Master: 6645111fc9e446e8877750a3c4779750 (127.0.99.136:44643)
I20240501 07:58:36.196142 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:46881
I20240501 07:58:36.422235 562 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:36.422940 825 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:36.423805 2142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:36.431020 2011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:36.434255 2273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:36.448284 692 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:36.460799 2404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:36.465569 955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:36.598681 2077 heartbeater.cc:503] Master 127.0.99.190:35609 was elected leader, sending a full tablet report...
I20240501 07:58:36.660095 436 ts_manager.cc:292] Set tserver state for 2c148dc68b5a4a38ba10be4e1e4b8cd2 to MAINTENANCE_MODE
W20240501 07:58:36.808141 2534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:36.808372 2534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 07:58:36.809316 2208 heartbeater.cc:503] Master 127.0.99.190:35609 was elected leader, sending a full tablet report...
W20240501 07:58:36.812203 2534 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 07:58:36.912387 825 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:36.915922 2404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:36.924464 2273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:36.928721 562 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:36.945011 2142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:36.945798 692 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:36.946053 955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:36.955024 2011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:36.987542 2273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:36.988624 825 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:36.991809 2142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:36.992760 955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:37.002151 562 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:37.009038 692 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:37.012648 2339 heartbeater.cc:503] Master 127.0.99.190:35609 was elected leader, sending a full tablet report...
I20240501 07:58:37.027155 2404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:37.039008 2011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:37.157204 825 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:37.161660 2404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:37.162009 2011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:37.170608 562 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:37.183130 2142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:37.192658 955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:37.198215 2470 heartbeater.cc:503] Master 127.0.99.190:35609 was elected leader, sending a full tablet report...
I20240501 07:58:37.199810 2273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 07:58:37.215468 2611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:37.215839 2611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:37.219873 2611 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 07:58:37.221506 692 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:37.328071 2011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:37.333251 2273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:37.338120 562 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:37.340103 2142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:37.352737 692 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:37.371492 2404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:37.374969 955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:37.380016 2142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:37.392617 562 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:37.397919 2011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:37.399163 825 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:37.405992 2273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:37.407284 825 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:37.421582 2404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:37.422612 955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:37.448953 692 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:37.486341 582 consensus_queue.cc:237] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:37.488337 975 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Refusing update from remote peer 2c148dc68b5a4a38ba10be4e1e4b8cd2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:37.488350 713 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Refusing update from remote peer 2c148dc68b5a4a38ba10be4e1e4b8cd2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:37.489094 2737 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:37.489594 1948 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:37.492504 1948 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } }
I20240501 07:58:37.493427 975 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } }
I20240501 07:58:37.495410 713 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } }
I20240501 07:58:37.496598 440 catalog_manager.cc:5484] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:37.502319 582 consensus_queue.cc:237] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } }
I20240501 07:58:37.504127 713 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Refusing update from remote peer 2c148dc68b5a4a38ba10be4e1e4b8cd2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:37.504597 975 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Refusing update from remote peer 2c148dc68b5a4a38ba10be4e1e4b8cd2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:37.504596 2741 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:37.504953 2741 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:37.506839 2737 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6645111fc9e446e8877750a3c4779750 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } } }
I20240501 07:58:37.507966 975 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6645111fc9e446e8877750a3c4779750 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } } }
I20240501 07:58:37.508513 426 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9d5700ab4d4f465bbe08ad25d0b9084f with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 07:58:37.508982 713 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6645111fc9e446e8877750a3c4779750 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } } }
I20240501 07:58:37.509578 432 catalog_manager.cc:5484] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 reported cstate change: config changed from index 4 to 5, NON_VOTER 6645111fc9e446e8877750a3c4779750 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:37.515075 518 consensus_peers.cc:485] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 -> Peer 6645111fc9e446e8877750a3c4779750 (127.0.99.136:44643): Couldn't send request to peer 6645111fc9e446e8877750a3c4779750. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d5700ab4d4f465bbe08ad25d0b9084f. This is attempt 1: this message will repeat every 5th retry.
W20240501 07:58:37.619606 2738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:37.619875 2738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:37.623903 2738 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 07:58:37.727010 825 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:37.727191 2142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:37.729610 2404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:37.735076 562 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:37.738720 2011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:37.745355 955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:37.745601 692 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:37.756095 2273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 07:58:37.941371 2794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:37.941713 2794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:37.946383 2794 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 07:58:38.045621 2142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:38.045850 562 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:38.056314 692 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:38.060214 825 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:38.063920 2011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:38.067588 2404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:38.071882 955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:38.072329 2273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:38.095172 2837 ts_tablet_manager.cc:902] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750: Initiating tablet copy from peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849)
I20240501 07:58:38.096089 2837 tablet_copy_client.cc:308] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:35849
I20240501 07:58:38.105760 603 tablet_copy_service.cc:140] P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Received BeginTabletCopySession request for tablet 9d5700ab4d4f465bbe08ad25d0b9084f from peer 6645111fc9e446e8877750a3c4779750 ({username='slave'} at 127.0.99.136:52589)
I20240501 07:58:38.105991 603 tablet_copy_service.cc:161] P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Beginning new tablet copy session on tablet 9d5700ab4d4f465bbe08ad25d0b9084f from peer 6645111fc9e446e8877750a3c4779750 at {username='slave'} at 127.0.99.136:52589: session id = 6645111fc9e446e8877750a3c4779750-9d5700ab4d4f465bbe08ad25d0b9084f
I20240501 07:58:38.107547 603 tablet_copy_source_session.cc:205] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:38.111356 2837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d5700ab4d4f465bbe08ad25d0b9084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:38.117961 2837 tablet_copy_client.cc:783] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:38.118183 2837 tablet_copy_client.cc:647] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:38.119990 2840 tablet_copy_client.cc:1002] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750: tablet copy: Injecting latency into file download: 1500
W20240501 07:58:38.252779 2843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:38.253064 2843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:38.256755 2843 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 07:58:38.354918 562 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:38.365029 2011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:38.367832 825 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:38.370534 692 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:38.371170 2142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:38.373189 2404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:38.374395 955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:38.396665 2273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240501 07:58:38.436787 2533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 501
W20240501 07:58:38.507828 648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
W20240501 07:58:38.508730 648 consensus_peers.cc:485] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:58:38.555332 779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
W20240501 07:58:38.556322 779 consensus_peers.cc:485] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:58:38.585917 648 consensus_peers.cc:485] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:58:38.590200 648 consensus_peers.cc:485] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:58:38.615561 910 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
W20240501 07:58:38.616513 910 consensus_peers.cc:485] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:58:38.623785 910 consensus_peers.cc:485] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:38.671095 2011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:38.671672 2142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:38.679411 692 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:38.680461 2273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:38.687216 2404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:38.687237 825 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:38.690997 955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:38.789742 825 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:38.795049 692 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:38.795296 2404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:38.795629 2273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:38.801753 2011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:38.817394 2142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:38.828617 955 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 07:58:38.829937 779 consensus_peers.cc:485] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:38.864002 975 consensus_queue.cc:237] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: true } }
I20240501 07:58:38.865805 713 raft_consensus.cc:1261] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:38.866003 843 raft_consensus.cc:1261] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:38.866494 2978 consensus_queue.cc:1035] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:38.868597 2519 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: true } } }
I20240501 07:58:38.870795 2836 consensus_queue.cc:1035] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:38.871125 843 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: true } } }
I20240501 07:58:38.871495 432 catalog_manager.cc:5484] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 reported cstate change: config changed from index -1 to 4, NON_VOTER b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "e2edc87d834f4e0bb662d501af69f2b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:38.873304 713 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: true } } }
W20240501 07:58:38.875996 909 consensus_peers.cc:485] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 -> Peer b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133:39953): Couldn't send request to peer b331ee755f7d4df48eb9f4460a0340b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0638f250a164460a694cd41b3e9f27d. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:38.883921 713 consensus_queue.cc:237] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } }
I20240501 07:58:38.885910 843 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:38.886471 975 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:38.886765 2991 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:38.886994 2654 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:38.888979 2894 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 36272be7c71b413a94d36b4257d34425 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } } }
I20240501 07:58:38.889394 843 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 36272be7c71b413a94d36b4257d34425 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } } }
I20240501 07:58:38.891309 975 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 36272be7c71b413a94d36b4257d34425 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } } }
W20240501 07:58:38.891464 649 consensus_peers.cc:485] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 36272be7c71b413a94d36b4257d34425 (127.0.99.134:39945): Couldn't send request to peer 36272be7c71b413a94d36b4257d34425. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 004e01dbd0f94cbeb20cba08164d386d. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:38.891779 440 catalog_manager.cc:5484] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: config changed from index -1 to 4, NON_VOTER 36272be7c71b413a94d36b4257d34425 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:38.941350 910 consensus_peers.cc:485] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:39.343040 3026 ts_tablet_manager.cc:902] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425: Initiating tablet copy from peer e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577)
I20240501 07:58:39.343730 3026 tablet_copy_client.cc:308] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43577
I20240501 07:58:39.347978 734 tablet_copy_service.cc:140] P e2c804beb8ef40ecb16c952f8875ca02: Received BeginTabletCopySession request for tablet 004e01dbd0f94cbeb20cba08164d386d from peer 36272be7c71b413a94d36b4257d34425 ({username='slave'} at 127.0.99.134:53971)
I20240501 07:58:39.348210 734 tablet_copy_service.cc:161] P e2c804beb8ef40ecb16c952f8875ca02: Beginning new tablet copy session on tablet 004e01dbd0f94cbeb20cba08164d386d from peer 36272be7c71b413a94d36b4257d34425 at {username='slave'} at 127.0.99.134:53971: session id = 36272be7c71b413a94d36b4257d34425-004e01dbd0f94cbeb20cba08164d386d
I20240501 07:58:39.349637 734 tablet_copy_source_session.cc:205] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:39.352941 3026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004e01dbd0f94cbeb20cba08164d386d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:39.359936 3026 tablet_copy_client.cc:783] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:39.360162 3026 tablet_copy_client.cc:647] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:39.361861 3029 tablet_copy_client.cc:1002] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425: tablet copy: Injecting latency into file download: 1500
I20240501 07:58:39.440627 3032 ts_tablet_manager.cc:902] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6: Initiating tablet copy from peer e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973)
I20240501 07:58:39.441408 3032 tablet_copy_client.cc:308] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36973
I20240501 07:58:39.445931 996 tablet_copy_service.cc:140] P e2edc87d834f4e0bb662d501af69f2b5: Received BeginTabletCopySession request for tablet e0638f250a164460a694cd41b3e9f27d from peer b331ee755f7d4df48eb9f4460a0340b6 ({username='slave'} at 127.0.99.133:42333)
I20240501 07:58:39.446110 996 tablet_copy_service.cc:161] P e2edc87d834f4e0bb662d501af69f2b5: Beginning new tablet copy session on tablet e0638f250a164460a694cd41b3e9f27d from peer b331ee755f7d4df48eb9f4460a0340b6 at {username='slave'} at 127.0.99.133:42333: session id = b331ee755f7d4df48eb9f4460a0340b6-e0638f250a164460a694cd41b3e9f27d
I20240501 07:58:39.447487 996 tablet_copy_source_session.cc:205] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:39.450917 3032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0638f250a164460a694cd41b3e9f27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:39.456661 3032 tablet_copy_client.cc:783] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:39.456890 3032 tablet_copy_client.cc:647] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:39.458766 3036 tablet_copy_client.cc:1002] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6: tablet copy: Injecting latency into file download: 1500
I20240501 07:58:39.537856 2978 raft_consensus.cc:482] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2c148dc68b5a4a38ba10be4e1e4b8cd2)
I20240501 07:58:39.537997 2978 raft_consensus.cc:504] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } }
I20240501 07:58:39.538381 2978 leader_election.cc:288] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849), e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577)
I20240501 07:58:39.538975 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5700ab4d4f465bbe08ad25d0b9084f" candidate_uuid: "e2edc87d834f4e0bb662d501af69f2b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02" is_pre_election: true
W20240501 07:58:39.539469 910 leader_election.cc:334] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
I20240501 07:58:39.539631 910 leader_election.cc:302] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2edc87d834f4e0bb662d501af69f2b5; no voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2, e2c804beb8ef40ecb16c952f8875ca02
I20240501 07:58:39.539824 2978 raft_consensus.cc:2726] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:58:39.598950 2991 raft_consensus.cc:482] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2c148dc68b5a4a38ba10be4e1e4b8cd2)
I20240501 07:58:39.599285 2991 raft_consensus.cc:504] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:39.600162 2991 leader_election.cc:288] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849), 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475)
I20240501 07:58:39.600936 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e4d59526a048fb8bb06d8e4a9de604" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4" is_pre_election: true
W20240501 07:58:39.601897 648 leader_election.cc:334] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
I20240501 07:58:39.602058 648 leader_election.cc:302] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2c804beb8ef40ecb16c952f8875ca02; no voters: 10aab83192f447829c9b8254e36d94b4, 2c148dc68b5a4a38ba10be4e1e4b8cd2
I20240501 07:58:39.602283 2991 raft_consensus.cc:2726] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:58:39.603227 2991 raft_consensus.cc:482] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2c148dc68b5a4a38ba10be4e1e4b8cd2)
I20240501 07:58:39.603327 2991 raft_consensus.cc:504] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } }
I20240501 07:58:39.603765 2991 leader_election.cc:288] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849), e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973)
I20240501 07:58:39.604265 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5700ab4d4f465bbe08ad25d0b9084f" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5" is_pre_election: true
I20240501 07:58:39.604416 975 raft_consensus.cc:2454] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 1.
W20240501 07:58:39.604847 648 leader_election.cc:334] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
I20240501 07:58:39.604950 649 leader_election.cc:302] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e2c804beb8ef40ecb16c952f8875ca02, e2edc87d834f4e0bb662d501af69f2b5; no voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2
I20240501 07:58:39.605146 2991 raft_consensus.cc:2781] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:58:39.605203 2991 raft_consensus.cc:482] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2c148dc68b5a4a38ba10be4e1e4b8cd2)
I20240501 07:58:39.605242 2991 raft_consensus.cc:3037] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:39.606134 2991 raft_consensus.cc:504] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } }
I20240501 07:58:39.606482 2991 leader_election.cc:288] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 2 election: Requested vote from peers 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849), e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973)
I20240501 07:58:39.606839 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5700ab4d4f465bbe08ad25d0b9084f" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5"
I20240501 07:58:39.606971 975 raft_consensus.cc:3037] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:58:39.607558 648 leader_election.cc:334] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
I20240501 07:58:39.607935 975 raft_consensus.cc:2454] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c804beb8ef40ecb16c952f8875ca02 in term 2.
I20240501 07:58:39.608211 649 leader_election.cc:302] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e2c804beb8ef40ecb16c952f8875ca02, e2edc87d834f4e0bb662d501af69f2b5; no voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2
I20240501 07:58:39.608376 2991 raft_consensus.cc:2781] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:39.608469 2991 raft_consensus.cc:686] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 2 LEADER]: Becoming Leader. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Running, Role: LEADER
I20240501 07:58:39.608601 2991 consensus_queue.cc:237] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } }
I20240501 07:58:39.610656 432 catalog_manager.cc:5484] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: term changed from 1 to 2, leader changed from 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129) to e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:39.620504 2837 tablet_copy_client.cc:523] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:39.622457 2837 tablet_bootstrap.cc:492] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750: Bootstrap starting.
I20240501 07:58:39.628870 2837 log.cc:824] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:39.653622 2837 tablet_bootstrap.cc:492] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:39.654002 2837 tablet_bootstrap.cc:492] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750: Bootstrap complete.
I20240501 07:58:39.654294 2837 ts_tablet_manager.cc:1367] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750: Time spent bootstrapping tablet: real 0.032s user 0.029s sys 0.000s
I20240501 07:58:39.656778 2837 raft_consensus.cc:348] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } }
I20240501 07:58:39.657058 2837 raft_consensus.cc:729] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6645111fc9e446e8877750a3c4779750, State: Initialized, Role: LEARNER
I20240501 07:58:39.657383 2837 consensus_queue.cc:260] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } }
I20240501 07:58:39.658190 2837 ts_tablet_manager.cc:1398] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
W20240501 07:58:39.659502 2837 tablet_copy_client.cc:1107] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
I20240501 07:58:39.678227 3014 raft_consensus.cc:482] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2c148dc68b5a4a38ba10be4e1e4b8cd2)
I20240501 07:58:39.678390 3014 raft_consensus.cc:504] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:39.678840 3014 leader_election.cc:288] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849), e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577)
I20240501 07:58:39.679652 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e4d59526a048fb8bb06d8e4a9de604" candidate_uuid: "10aab83192f447829c9b8254e36d94b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02" is_pre_election: true
I20240501 07:58:39.679805 713 raft_consensus.cc:2454] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10aab83192f447829c9b8254e36d94b4 in term 1.
I20240501 07:58:39.680114 779 leader_election.cc:302] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10aab83192f447829c9b8254e36d94b4, e2c804beb8ef40ecb16c952f8875ca02; no voters:
W20240501 07:58:39.680275 779 leader_election.cc:334] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
I20240501 07:58:39.680354 3014 raft_consensus.cc:2781] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:58:39.680434 3014 raft_consensus.cc:482] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2c148dc68b5a4a38ba10be4e1e4b8cd2)
I20240501 07:58:39.680531 3014 raft_consensus.cc:3037] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:39.681831 3014 raft_consensus.cc:504] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:39.682691 3014 leader_election.cc:288] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 2 election: Requested vote from peers 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849), e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577)
I20240501 07:58:39.683265 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e4d59526a048fb8bb06d8e4a9de604" candidate_uuid: "10aab83192f447829c9b8254e36d94b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02"
I20240501 07:58:39.683411 713 raft_consensus.cc:3037] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:58:39.683741 779 leader_election.cc:334] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
I20240501 07:58:39.684376 713 raft_consensus.cc:2454] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10aab83192f447829c9b8254e36d94b4 in term 2.
I20240501 07:58:39.684762 779 leader_election.cc:302] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10aab83192f447829c9b8254e36d94b4, e2c804beb8ef40ecb16c952f8875ca02; no voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2
I20240501 07:58:39.685005 3014 raft_consensus.cc:2781] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:39.685113 3014 raft_consensus.cc:686] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [term 2 LEADER]: Becoming Leader. State: Replica: 10aab83192f447829c9b8254e36d94b4, State: Running, Role: LEADER
I20240501 07:58:39.685531 3014 consensus_queue.cc:237] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:39.687665 432 catalog_manager.cc:5484] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 reported cstate change: term changed from 1 to 2, leader changed from 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129) to 10aab83192f447829c9b8254e36d94b4 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "10aab83192f447829c9b8254e36d94b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:39.734549 2991 raft_consensus.cc:482] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2c148dc68b5a4a38ba10be4e1e4b8cd2)
I20240501 07:58:39.734665 2991 raft_consensus.cc:504] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:39.735054 2991 leader_election.cc:288] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475), 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849)
I20240501 07:58:39.735523 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f220c7dad9d4cf390608b13c5de0fbf" candidate_uuid: "e2c804beb8ef40ecb16c952f8875ca02" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4" is_pre_election: true
W20240501 07:58:39.736455 648 leader_election.cc:334] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
I20240501 07:58:39.736527 648 leader_election.cc:302] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2c804beb8ef40ecb16c952f8875ca02; no voters: 10aab83192f447829c9b8254e36d94b4, 2c148dc68b5a4a38ba10be4e1e4b8cd2
I20240501 07:58:39.736689 2991 raft_consensus.cc:2726] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:58:39.884268 3014 raft_consensus.cc:482] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2c148dc68b5a4a38ba10be4e1e4b8cd2)
I20240501 07:58:39.884421 3014 raft_consensus.cc:504] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:39.885032 3014 leader_election.cc:288] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973), 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849)
I20240501 07:58:39.885692 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ad498fe543450f8911b83dd71d07f3" candidate_uuid: "10aab83192f447829c9b8254e36d94b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5" is_pre_election: true
W20240501 07:58:39.886495 779 leader_election.cc:334] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
I20240501 07:58:39.886618 779 leader_election.cc:302] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10aab83192f447829c9b8254e36d94b4; no voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2, e2edc87d834f4e0bb662d501af69f2b5
I20240501 07:58:39.886837 3014 raft_consensus.cc:2726] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:58:39.910672 2978 raft_consensus.cc:482] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2c148dc68b5a4a38ba10be4e1e4b8cd2)
I20240501 07:58:39.910821 2978 raft_consensus.cc:504] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:39.911259 2978 leader_election.cc:288] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475), 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849)
I20240501 07:58:39.911751 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ad498fe543450f8911b83dd71d07f3" candidate_uuid: "e2edc87d834f4e0bb662d501af69f2b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4" is_pre_election: true
I20240501 07:58:39.911913 843 raft_consensus.cc:2454] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2edc87d834f4e0bb662d501af69f2b5 in term 1.
I20240501 07:58:39.912436 909 leader_election.cc:302] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10aab83192f447829c9b8254e36d94b4, e2edc87d834f4e0bb662d501af69f2b5; no voters:
I20240501 07:58:39.912801 2978 raft_consensus.cc:2781] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240501 07:58:39.912838 910 leader_election.cc:334] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
I20240501 07:58:39.912905 2978 raft_consensus.cc:482] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2c148dc68b5a4a38ba10be4e1e4b8cd2)
I20240501 07:58:39.913069 2978 raft_consensus.cc:3037] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:39.914134 2978 raft_consensus.cc:504] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:39.914527 2978 leader_election.cc:288] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 2 election: Requested vote from peers 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475), 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849)
I20240501 07:58:39.914961 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ad498fe543450f8911b83dd71d07f3" candidate_uuid: "e2edc87d834f4e0bb662d501af69f2b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "10aab83192f447829c9b8254e36d94b4"
I20240501 07:58:39.915110 843 raft_consensus.cc:3037] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:58:39.915890 910 leader_election.cc:334] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
I20240501 07:58:39.916224 843 raft_consensus.cc:2454] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2edc87d834f4e0bb662d501af69f2b5 in term 2.
I20240501 07:58:39.916643 909 leader_election.cc:302] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10aab83192f447829c9b8254e36d94b4, e2edc87d834f4e0bb662d501af69f2b5; no voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2
I20240501 07:58:39.916823 2978 raft_consensus.cc:2781] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:39.917039 2978 raft_consensus.cc:686] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [term 2 LEADER]: Becoming Leader. State: Replica: e2edc87d834f4e0bb662d501af69f2b5, State: Running, Role: LEADER
I20240501 07:58:39.917371 2978 consensus_queue.cc:237] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:39.919498 432 catalog_manager.cc:5484] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 reported cstate change: term changed from 1 to 2, leader changed from 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129) to e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "e2edc87d834f4e0bb662d501af69f2b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:39.931435 3014 raft_consensus.cc:482] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2c148dc68b5a4a38ba10be4e1e4b8cd2)
I20240501 07:58:39.931567 3014 raft_consensus.cc:504] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:39.931977 3014 leader_election.cc:288] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577), 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849)
I20240501 07:58:39.932354 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f220c7dad9d4cf390608b13c5de0fbf" candidate_uuid: "10aab83192f447829c9b8254e36d94b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02" is_pre_election: true
I20240501 07:58:39.932511 713 raft_consensus.cc:2454] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10aab83192f447829c9b8254e36d94b4 in term 1.
I20240501 07:58:39.933272 779 leader_election.cc:302] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10aab83192f447829c9b8254e36d94b4, e2c804beb8ef40ecb16c952f8875ca02; no voters:
I20240501 07:58:39.933535 3014 raft_consensus.cc:2781] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240501 07:58:39.933535 779 leader_election.cc:334] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
I20240501 07:58:39.933614 3014 raft_consensus.cc:482] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2c148dc68b5a4a38ba10be4e1e4b8cd2)
I20240501 07:58:39.933775 3014 raft_consensus.cc:3037] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:39.934828 3014 raft_consensus.cc:504] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:39.935276 3014 leader_election.cc:288] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 2 election: Requested vote from peers e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577), 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849)
I20240501 07:58:39.935856 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f220c7dad9d4cf390608b13c5de0fbf" candidate_uuid: "10aab83192f447829c9b8254e36d94b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02"
I20240501 07:58:39.935997 713 raft_consensus.cc:3037] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:58:39.936764 779 leader_election.cc:334] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
I20240501 07:58:39.936978 713 raft_consensus.cc:2454] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10aab83192f447829c9b8254e36d94b4 in term 2.
I20240501 07:58:39.937315 779 leader_election.cc:302] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10aab83192f447829c9b8254e36d94b4, e2c804beb8ef40ecb16c952f8875ca02; no voters: 2c148dc68b5a4a38ba10be4e1e4b8cd2
I20240501 07:58:39.937494 3014 raft_consensus.cc:2781] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:39.937884 3014 raft_consensus.cc:686] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [term 2 LEADER]: Becoming Leader. State: Replica: 10aab83192f447829c9b8254e36d94b4, State: Running, Role: LEADER
I20240501 07:58:39.938041 3014 consensus_queue.cc:237] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:39.940295 432 catalog_manager.cc:5484] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 reported cstate change: term changed from 1 to 2, leader changed from 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129) to 10aab83192f447829c9b8254e36d94b4 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "10aab83192f447829c9b8254e36d94b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:40.064339 2424 raft_consensus.cc:3037] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 1 LEARNER]: Advancing to term 2
I20240501 07:58:40.065529 2424 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 2 LEARNER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:58:40.066368 2654 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:40.068207 975 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:58:40.068733 2654 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:40.069645 2654 raft_consensus.cc:1050] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02: attempting to promote NON_VOTER 6645111fc9e446e8877750a3c4779750 to VOTER
W20240501 07:58:40.069787 2654 raft_consensus.cc:1053] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02: Unable to promote non-voter 6645111fc9e446e8877750a3c4779750: Illegal state: Leader has not yet committed an operation in its own term
W20240501 07:58:40.072402 648 consensus_peers.cc:485] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:40.072475 2991 raft_consensus.cc:1050] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02: attempting to promote NON_VOTER 6645111fc9e446e8877750a3c4779750 to VOTER
I20240501 07:58:40.072885 2991 consensus_queue.cc:237] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } }
I20240501 07:58:40.074538 976 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 07:58:40.074867 2424 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 2 LEARNER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 07:58:40.075241 3084 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 07:58:40.075767 648 consensus_peers.cc:485] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:40.076326 3084 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:40.078640 3066 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 6645111fc9e446e8877750a3c4779750 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
I20240501 07:58:40.079066 975 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 6645111fc9e446e8877750a3c4779750 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
I20240501 07:58:40.080916 2424 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 6645111fc9e446e8877750a3c4779750 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
I20240501 07:58:40.081552 432 catalog_manager.cc:5484] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: config changed from index 5 to 7, 6645111fc9e446e8877750a3c4779750 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:40.084956 713 consensus_queue.cc:237] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } }
I20240501 07:58:40.085716 714 raft_consensus.cc:1261] T d6e4d59526a048fb8bb06d8e4a9de604 P e2c804beb8ef40ecb16c952f8875ca02 [term 2 FOLLOWER]: Refusing update from remote peer 10aab83192f447829c9b8254e36d94b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 07:58:40.086488 3014 consensus_queue.cc:1035] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:40.086820 975 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 07:58:40.087245 2424 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 07:58:40.087538 3084 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:58:40.087708 2654 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:58:40.089316 2654 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
W20240501 07:58:40.090560 779 consensus_peers.cc:485] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:40.091217 426 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d5700ab4d4f465bbe08ad25d0b9084f with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:58:40.092267 438 catalog_manager.cc:5484] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: config changed from index 7 to 8, VOTER 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:40.092475 2424 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
I20240501 07:58:40.092918 975 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
W20240501 07:58:40.097029 426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
W20240501 07:58:40.098140 426 catalog_manager.cc:4573] TS 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d5700ab4d4f465bbe08ad25d0b9084f: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
I20240501 07:58:40.329499 714 raft_consensus.cc:1261] T 4f220c7dad9d4cf390608b13c5de0fbf P e2c804beb8ef40ecb16c952f8875ca02 [term 2 FOLLOWER]: Refusing update from remote peer 10aab83192f447829c9b8254e36d94b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 07:58:40.330054 3088 consensus_queue.cc:1035] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:58:40.333865 779 consensus_peers.cc:485] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:40.334623 843 raft_consensus.cc:1261] T 46ad498fe543450f8911b83dd71d07f3 P 10aab83192f447829c9b8254e36d94b4 [term 2 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 07:58:40.335188 2836 consensus_queue.cc:1035] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:58:40.339326 910 consensus_peers.cc:485] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:58:40.852922 648 consensus_peers.cc:485] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 07:58:40.862337 3026 tablet_copy_client.cc:523] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:40.864308 3026 tablet_bootstrap.cc:492] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425: Bootstrap starting.
I20240501 07:58:40.871202 3026 log.cc:824] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:40.893684 3026 tablet_bootstrap.cc:492] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:40.894198 3026 tablet_bootstrap.cc:492] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425: Bootstrap complete.
I20240501 07:58:40.894582 3026 ts_tablet_manager.cc:1367] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.010s
I20240501 07:58:40.896624 3026 raft_consensus.cc:348] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } }
I20240501 07:58:40.896862 3026 raft_consensus.cc:729] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 36272be7c71b413a94d36b4257d34425, State: Initialized, Role: LEARNER
I20240501 07:58:40.897156 3026 consensus_queue.cc:260] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } }
I20240501 07:58:40.897938 3026 ts_tablet_manager.cc:1398] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 07:58:40.898790 734 tablet_copy_service.cc:342] P e2c804beb8ef40ecb16c952f8875ca02: Request end of tablet copy session 36272be7c71b413a94d36b4257d34425-004e01dbd0f94cbeb20cba08164d386d received from {username='slave'} at 127.0.99.134:53971
I20240501 07:58:40.898885 734 tablet_copy_service.cc:433] P e2c804beb8ef40ecb16c952f8875ca02: ending tablet copy session 36272be7c71b413a94d36b4257d34425-004e01dbd0f94cbeb20cba08164d386d on tablet 004e01dbd0f94cbeb20cba08164d386d with peer 36272be7c71b413a94d36b4257d34425
W20240501 07:58:40.929798 648 consensus_peers.cc:485] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 07:58:40.959308 3032 tablet_copy_client.cc:523] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:40.961372 3032 tablet_bootstrap.cc:492] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6: Bootstrap starting.
I20240501 07:58:40.966729 3032 log.cc:824] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:40.987988 3032 tablet_bootstrap.cc:492] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:40.988355 3032 tablet_bootstrap.cc:492] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6: Bootstrap complete.
I20240501 07:58:40.988667 3032 ts_tablet_manager.cc:1367] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6: Time spent bootstrapping tablet: real 0.027s user 0.019s sys 0.008s
I20240501 07:58:40.990981 3032 raft_consensus.cc:348] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: true } }
I20240501 07:58:40.991222 3032 raft_consensus.cc:729] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b331ee755f7d4df48eb9f4460a0340b6, State: Initialized, Role: LEARNER
I20240501 07:58:40.991551 3032 consensus_queue.cc:260] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: true } }
I20240501 07:58:40.992410 3032 ts_tablet_manager.cc:1398] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 07:58:40.993279 996 tablet_copy_service.cc:342] P e2edc87d834f4e0bb662d501af69f2b5: Request end of tablet copy session b331ee755f7d4df48eb9f4460a0340b6-e0638f250a164460a694cd41b3e9f27d received from {username='slave'} at 127.0.99.133:42333
I20240501 07:58:40.993366 996 tablet_copy_service.cc:433] P e2edc87d834f4e0bb662d501af69f2b5: ending tablet copy session b331ee755f7d4df48eb9f4460a0340b6-e0638f250a164460a694cd41b3e9f27d on tablet e0638f250a164460a694cd41b3e9f27d with peer b331ee755f7d4df48eb9f4460a0340b6
I20240501 07:58:41.011586 2162 raft_consensus.cc:1203] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 07:58:41.014304 910 consensus_peers.cc:485] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:58:41.041210 910 consensus_peers.cc:485] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:58:41.046178 779 consensus_peers.cc:485] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 07:58:41.065670 2014 raft_consensus.cc:1203] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 07:58:41.243877 648 consensus_peers.cc:485] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:58:41.311689 426 catalog_manager.cc:4573] TS 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d5700ab4d4f465bbe08ad25d0b9084f: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
W20240501 07:58:41.319893 779 consensus_peers.cc:485] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:58:41.325955 910 consensus_peers.cc:485] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 07:58:41.571926 3169 raft_consensus.cc:1050] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5: attempting to promote NON_VOTER b331ee755f7d4df48eb9f4460a0340b6 to VOTER
I20240501 07:58:41.572526 3169 consensus_queue.cc:237] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } }
I20240501 07:58:41.573947 714 raft_consensus.cc:1261] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:41.574118 2014 raft_consensus.cc:1261] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6 [term 1 LEARNER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:41.574204 843 raft_consensus.cc:1261] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:41.574383 3169 consensus_queue.cc:1035] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:41.574787 3170 consensus_queue.cc:1035] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:41.575058 3172 consensus_queue.cc:1035] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:41.578629 3170 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } }
I20240501 07:58:41.579350 843 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } }
I20240501 07:58:41.579416 2014 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } }
I20240501 07:58:41.581864 440 catalog_manager.cc:5484] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 reported cstate change: config changed from index 4 to 5, b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e2edc87d834f4e0bb662d501af69f2b5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } }
I20240501 07:58:41.586062 975 consensus_queue.cc:237] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } }
I20240501 07:58:41.586882 714 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } }
I20240501 07:58:41.587340 2014 raft_consensus.cc:1261] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:58:41.587469 843 raft_consensus.cc:1261] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:58:41.587735 2519 consensus_queue.cc:1035] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:41.588308 2519 consensus_queue.cc:1035] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:41.589994 3169 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } }
I20240501 07:58:41.590838 843 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } }
I20240501 07:58:41.591571 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e0638f250a164460a694cd41b3e9f27d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:58:41.591517 2014 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } }
I20240501 07:58:41.592474 432 catalog_manager.cc:5484] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 reported cstate change: config changed from index 5 to 6, VOTER e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "e2edc87d834f4e0bb662d501af69f2b5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:41.595397 692 tablet_service.cc:1514] Processing DeleteTablet for tablet e0638f250a164460a694cd41b3e9f27d with delete_type TABLET_DATA_TOMBSTONED (TS e2c804beb8ef40ecb16c952f8875ca02 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43240
I20240501 07:58:41.595891 3181 tablet_replica.cc:330] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02: stopping tablet replica
I20240501 07:58:41.596236 3181 raft_consensus.cc:2229] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:41.596447 3181 raft_consensus.cc:2258] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.597492 3181 ts_tablet_manager.cc:1875] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:41.599494 3181 ts_tablet_manager.cc:1888] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:58:41.599575 3181 log.cc:1196] T e0638f250a164460a694cd41b3e9f27d P e2c804beb8ef40ecb16c952f8875ca02: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e0638f250a164460a694cd41b3e9f27d
I20240501 07:58:41.600374 426 catalog_manager.cc:4830] TS e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577): tablet e0638f250a164460a694cd41b3e9f27d (table rebalance_test_table_1 [id=e08d5aefd3334c55a539f0f47df899e4]) successfully deleted
I20240501 07:58:41.615726 2991 raft_consensus.cc:1050] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02: attempting to promote NON_VOTER 36272be7c71b413a94d36b4257d34425 to VOTER
I20240501 07:58:41.616218 2991 consensus_queue.cc:237] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } }
I20240501 07:58:41.617988 843 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:41.618162 2162 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 1 LEARNER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:41.618147 976 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:41.618391 3185 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:41.618714 3182 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:41.618980 2991 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:41.621191 3185 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 36272be7c71b413a94d36b4257d34425 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:41.621644 843 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 36272be7c71b413a94d36b4257d34425 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:41.623457 976 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 36272be7c71b413a94d36b4257d34425 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:41.624331 438 catalog_manager.cc:5484] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: config changed from index 4 to 5, 36272be7c71b413a94d36b4257d34425 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:41.626991 2162 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 36272be7c71b413a94d36b4257d34425 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:41.629992 714 consensus_queue.cc:237] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } }
I20240501 07:58:41.631369 976 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:58:41.631377 2162 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:58:41.631793 3184 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:41.632014 3185 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:41.633692 3185 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10aab83192f447829c9b8254e36d94b4 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:41.634253 976 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10aab83192f447829c9b8254e36d94b4 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:41.635262 426 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 004e01dbd0f94cbeb20cba08164d386d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:58:41.636384 432 catalog_manager.cc:5484] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: config changed from index 5 to 6, VOTER 10aab83192f447829c9b8254e36d94b4 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:41.636384 2162 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10aab83192f447829c9b8254e36d94b4 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:41.639111 825 tablet_service.cc:1514] Processing DeleteTablet for tablet 004e01dbd0f94cbeb20cba08164d386d with delete_type TABLET_DATA_TOMBSTONED (TS 10aab83192f447829c9b8254e36d94b4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50386
I20240501 07:58:41.640380 3190 tablet_replica.cc:330] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4: stopping tablet replica
I20240501 07:58:41.640753 3190 raft_consensus.cc:2229] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:41.641045 3190 raft_consensus.cc:2258] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:41.642292 3190 ts_tablet_manager.cc:1875] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:41.644229 3190 ts_tablet_manager.cc:1888] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:58:41.644320 3190 log.cc:1196] T 004e01dbd0f94cbeb20cba08164d386d P 10aab83192f447829c9b8254e36d94b4: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/004e01dbd0f94cbeb20cba08164d386d
I20240501 07:58:41.644989 425 catalog_manager.cc:4830] TS 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475): tablet 004e01dbd0f94cbeb20cba08164d386d (table rebalance_test_table_3 [id=0678a1a47ecc4cb3bb664b77ec709a59]) successfully deleted
I20240501 07:58:41.713196 714 consensus_queue.cc:237] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } }
I20240501 07:58:41.714778 843 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:58:41.714836 976 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:58:41.714977 649 consensus_peers.cc:485] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 6645111fc9e446e8877750a3c4779750 (127.0.99.136:44643): Couldn't send request to peer 6645111fc9e446e8877750a3c4779750. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8e9192810694fa1bf54ca88fb619d11. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.715191 2991 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:41.715401 3184 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:58:41.718153 3185 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6645111fc9e446e8877750a3c4779750 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } } }
I20240501 07:58:41.718875 976 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6645111fc9e446e8877750a3c4779750 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } } }
I20240501 07:58:41.719074 843 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6645111fc9e446e8877750a3c4779750 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } } }
I20240501 07:58:41.721432 438 catalog_manager.cc:5484] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: config changed from index -1 to 4, NON_VOTER 6645111fc9e446e8877750a3c4779750 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:42.095363 3219 ts_tablet_manager.cc:902] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750: Initiating tablet copy from peer e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577)
I20240501 07:58:42.095839 3219 tablet_copy_client.cc:308] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43577
I20240501 07:58:42.099557 734 tablet_copy_service.cc:140] P e2c804beb8ef40ecb16c952f8875ca02: Received BeginTabletCopySession request for tablet b8e9192810694fa1bf54ca88fb619d11 from peer 6645111fc9e446e8877750a3c4779750 ({username='slave'} at 127.0.99.136:35421)
I20240501 07:58:42.099674 734 tablet_copy_service.cc:161] P e2c804beb8ef40ecb16c952f8875ca02: Beginning new tablet copy session on tablet b8e9192810694fa1bf54ca88fb619d11 from peer 6645111fc9e446e8877750a3c4779750 at {username='slave'} at 127.0.99.136:35421: session id = 6645111fc9e446e8877750a3c4779750-b8e9192810694fa1bf54ca88fb619d11
I20240501 07:58:42.100430 734 tablet_copy_source_session.cc:205] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:42.101291 3219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8e9192810694fa1bf54ca88fb619d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:42.103160 3219 tablet_copy_client.cc:783] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:42.103319 3219 tablet_copy_client.cc:647] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:42.104506 3223 tablet_copy_client.cc:1002] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750: tablet copy: Injecting latency into file download: 1500
W20240501 07:58:42.380771 426 catalog_manager.cc:4573] TS 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d5700ab4d4f465bbe08ad25d0b9084f: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
W20240501 07:58:42.519985 910 consensus_peers.cc:485] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:58:42.833150 779 consensus_peers.cc:485] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:58:43.172113 779 consensus_peers.cc:485] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:58:43.218259 648 consensus_peers.cc:485] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:58:43.471127 910 consensus_peers.cc:485] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:58:43.533496 648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111) [suppressed 36 similar messages]
W20240501 07:58:43.534320 648 consensus_peers.cc:485] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:58:43.543252 779 consensus_peers.cc:485] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 07:58:43.604888 3219 tablet_copy_client.cc:523] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:43.607014 3219 tablet_bootstrap.cc:492] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750: Bootstrap starting.
I20240501 07:58:43.627733 3219 tablet_bootstrap.cc:492] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:43.628106 3219 tablet_bootstrap.cc:492] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750: Bootstrap complete.
I20240501 07:58:43.628304 3219 ts_tablet_manager.cc:1367] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750: Time spent bootstrapping tablet: real 0.021s user 0.020s sys 0.000s
I20240501 07:58:43.628878 3219 raft_consensus.cc:348] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } }
I20240501 07:58:43.629009 3219 raft_consensus.cc:729] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6645111fc9e446e8877750a3c4779750, State: Initialized, Role: LEARNER
I20240501 07:58:43.629117 3219 consensus_queue.cc:260] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } }
I20240501 07:58:43.629531 3219 ts_tablet_manager.cc:1398] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 07:58:43.630414 734 tablet_copy_service.cc:342] P e2c804beb8ef40ecb16c952f8875ca02: Request end of tablet copy session 6645111fc9e446e8877750a3c4779750-b8e9192810694fa1bf54ca88fb619d11 received from {username='slave'} at 127.0.99.136:35421
I20240501 07:58:43.630589 734 tablet_copy_service.cc:433] P e2c804beb8ef40ecb16c952f8875ca02: ending tablet copy session 6645111fc9e446e8877750a3c4779750-b8e9192810694fa1bf54ca88fb619d11 on tablet b8e9192810694fa1bf54ca88fb619d11 with peer 6645111fc9e446e8877750a3c4779750
W20240501 07:58:43.641678 910 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 07:58:43.642691 910 consensus_peers.cc:485] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:58:43.660369 779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 07:58:43.664429 910 consensus_peers.cc:485] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:58:43.679692 779 consensus_peers.cc:485] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:58:43.960691 648 consensus_peers.cc:485] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 07:58:44.067982 2424 raft_consensus.cc:1203] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 07:58:44.480356 426 catalog_manager.cc:4573] TS 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d5700ab4d4f465bbe08ad25d0b9084f: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
I20240501 07:58:44.519917 3295 raft_consensus.cc:1050] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02: attempting to promote NON_VOTER 6645111fc9e446e8877750a3c4779750 to VOTER
I20240501 07:58:44.520319 3295 consensus_queue.cc:237] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } }
I20240501 07:58:44.521698 843 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:44.521711 976 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:44.521719 2424 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 1 LEARNER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:44.522049 3295 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:44.522251 3297 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:44.522428 3294 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:44.524381 3295 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6645111fc9e446e8877750a3c4779750 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
I20240501 07:58:44.525065 2424 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6645111fc9e446e8877750a3c4779750 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
I20240501 07:58:44.525107 843 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6645111fc9e446e8877750a3c4779750 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
I20240501 07:58:44.526388 976 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6645111fc9e446e8877750a3c4779750 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
I20240501 07:58:44.527809 440 catalog_manager.cc:5484] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: config changed from index 4 to 5, 6645111fc9e446e8877750a3c4779750 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:44.531714 714 consensus_queue.cc:237] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } }
I20240501 07:58:44.532953 976 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:58:44.532945 2424 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:58:44.533331 3254 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:44.533545 3297 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:44.535200 3254 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10aab83192f447829c9b8254e36d94b4 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
I20240501 07:58:44.535943 2424 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10aab83192f447829c9b8254e36d94b4 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
I20240501 07:58:44.535959 976 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10aab83192f447829c9b8254e36d94b4 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
I20240501 07:58:44.537115 426 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b8e9192810694fa1bf54ca88fb619d11 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:58:44.537951 438 catalog_manager.cc:5484] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: config changed from index 5 to 6, VOTER 10aab83192f447829c9b8254e36d94b4 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:44.540628 825 tablet_service.cc:1514] Processing DeleteTablet for tablet b8e9192810694fa1bf54ca88fb619d11 with delete_type TABLET_DATA_TOMBSTONED (TS 10aab83192f447829c9b8254e36d94b4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50386
I20240501 07:58:44.541762 3305 tablet_replica.cc:330] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4: stopping tablet replica
I20240501 07:58:44.541985 3305 raft_consensus.cc:2229] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.542330 3305 raft_consensus.cc:2258] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.543139 3305 ts_tablet_manager.cc:1875] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:44.545099 3305 ts_tablet_manager.cc:1888] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:58:44.545171 3305 log.cc:1196] T b8e9192810694fa1bf54ca88fb619d11 P 10aab83192f447829c9b8254e36d94b4: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b8e9192810694fa1bf54ca88fb619d11
I20240501 07:58:44.545724 425 catalog_manager.cc:4830] TS 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475): tablet b8e9192810694fa1bf54ca88fb619d11 (table rebalance_test_table_1 [id=e08d5aefd3334c55a539f0f47df899e4]) successfully deleted
I20240501 07:58:44.693989 2404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:44.695605 2142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:44.696429 692 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:58:44.699946 825 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:44.708647 2011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:44.722159 2273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:44.729190 955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 07:58:44.756927 910 consensus_peers.cc:485] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 07:58:44.764312 714 consensus_queue.cc:237] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } }
I20240501 07:58:44.765978 2424 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 07:58:44.766388 3254 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:58:44.766665 976 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 07:58:44.767846 3254 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:58:44.769050 3254 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER c1b68243a7144af5835b051aa9916d13 (127.0.99.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } } }
I20240501 07:58:44.771571 2424 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER c1b68243a7144af5835b051aa9916d13 (127.0.99.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } } }
I20240501 07:58:44.771787 438 catalog_manager.cc:5484] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: config changed from index 8 to 9, NON_VOTER c1b68243a7144af5835b051aa9916d13 (127.0.99.135) added. New cstate: current_term: 2 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:44.771955 649 consensus_peers.cc:485] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 -> Peer c1b68243a7144af5835b051aa9916d13 (127.0.99.135:44501): Couldn't send request to peer c1b68243a7144af5835b051aa9916d13. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d5700ab4d4f465bbe08ad25d0b9084f. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:44.776160 976 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER c1b68243a7144af5835b051aa9916d13 (127.0.99.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } } }
I20240501 07:58:44.782164 714 consensus_queue.cc:237] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } }
I20240501 07:58:44.783648 2162 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:58:44.783694 976 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 07:58:44.783758 649 consensus_peers.cc:485] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 6645111fc9e446e8877750a3c4779750 (127.0.99.136:44643): Couldn't send request to peer 6645111fc9e446e8877750a3c4779750. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 004e01dbd0f94cbeb20cba08164d386d. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:44.784077 3295 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:58:44.784385 3295 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:58:44.786505 3254 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6645111fc9e446e8877750a3c4779750 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } } }
I20240501 07:58:44.786945 2162 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6645111fc9e446e8877750a3c4779750 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } } }
I20240501 07:58:44.787097 976 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6645111fc9e446e8877750a3c4779750 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } } }
I20240501 07:58:44.789829 440 catalog_manager.cc:5484] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 reported cstate change: config changed from index 6 to 7, NON_VOTER 6645111fc9e446e8877750a3c4779750 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } } }
I20240501 07:58:45.171898 3387 ts_tablet_manager.cc:902] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13: Initiating tablet copy from peer e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577)
I20240501 07:58:45.172580 3387 tablet_copy_client.cc:308] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43577
I20240501 07:58:45.176249 734 tablet_copy_service.cc:140] P e2c804beb8ef40ecb16c952f8875ca02: Received BeginTabletCopySession request for tablet 9d5700ab4d4f465bbe08ad25d0b9084f from peer c1b68243a7144af5835b051aa9916d13 ({username='slave'} at 127.0.99.135:33175)
I20240501 07:58:45.176378 734 tablet_copy_service.cc:161] P e2c804beb8ef40ecb16c952f8875ca02: Beginning new tablet copy session on tablet 9d5700ab4d4f465bbe08ad25d0b9084f from peer c1b68243a7144af5835b051aa9916d13 at {username='slave'} at 127.0.99.135:33175: session id = c1b68243a7144af5835b051aa9916d13-9d5700ab4d4f465bbe08ad25d0b9084f
I20240501 07:58:45.177129 734 tablet_copy_source_session.cc:205] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:45.180141 3387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d5700ab4d4f465bbe08ad25d0b9084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:45.185598 3387 tablet_copy_client.cc:783] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:45.185941 3387 tablet_copy_client.cc:647] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:45.187779 3391 tablet_copy_client.cc:1002] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13: tablet copy: Injecting latency into file download: 1500
I20240501 07:58:45.352651 3403 ts_tablet_manager.cc:902] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750: Initiating tablet copy from peer e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577)
I20240501 07:58:45.353231 3403 tablet_copy_client.cc:308] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43577
I20240501 07:58:45.353776 734 tablet_copy_service.cc:140] P e2c804beb8ef40ecb16c952f8875ca02: Received BeginTabletCopySession request for tablet 004e01dbd0f94cbeb20cba08164d386d from peer 6645111fc9e446e8877750a3c4779750 ({username='slave'} at 127.0.99.136:35421)
I20240501 07:58:45.353897 734 tablet_copy_service.cc:161] P e2c804beb8ef40ecb16c952f8875ca02: Beginning new tablet copy session on tablet 004e01dbd0f94cbeb20cba08164d386d from peer 6645111fc9e446e8877750a3c4779750 at {username='slave'} at 127.0.99.136:35421: session id = 6645111fc9e446e8877750a3c4779750-004e01dbd0f94cbeb20cba08164d386d
I20240501 07:58:45.354907 734 tablet_copy_source_session.cc:205] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:45.355923 3403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004e01dbd0f94cbeb20cba08164d386d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:45.358311 3403 tablet_copy_client.cc:783] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:45.358920 3403 tablet_copy_client.cc:647] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:45.365511 3407 tablet_copy_client.cc:1002] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750: tablet copy: Injecting latency into file download: 1500
W20240501 07:58:45.481410 779 consensus_peers.cc:485] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:58:45.604458 648 consensus_peers.cc:485] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:58:45.655930 779 consensus_peers.cc:485] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:58:45.963760 910 consensus_peers.cc:485] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:58:45.977208 648 consensus_peers.cc:485] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:58:46.019850 779 consensus_peers.cc:485] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:58:46.139078 910 consensus_peers.cc:485] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:58:46.193274 779 consensus_peers.cc:485] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:58:46.253836 910 consensus_peers.cc:485] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 07:58:46.688256 3387 tablet_copy_client.cc:523] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:46.690120 3387 tablet_bootstrap.cc:492] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13: Bootstrap starting.
I20240501 07:58:46.696074 3387 log.cc:824] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:46.720867 3387 tablet_bootstrap.cc:492] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:46.721377 3387 tablet_bootstrap.cc:492] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13: Bootstrap complete.
I20240501 07:58:46.721745 3387 ts_tablet_manager.cc:1367] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13: Time spent bootstrapping tablet: real 0.032s user 0.033s sys 0.000s
I20240501 07:58:46.724292 3387 raft_consensus.cc:348] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } }
I20240501 07:58:46.724584 3387 raft_consensus.cc:729] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c1b68243a7144af5835b051aa9916d13, State: Initialized, Role: LEARNER
I20240501 07:58:46.725023 3387 consensus_queue.cc:260] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } }
I20240501 07:58:46.725970 3387 ts_tablet_manager.cc:1398] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 07:58:46.727035 734 tablet_copy_service.cc:342] P e2c804beb8ef40ecb16c952f8875ca02: Request end of tablet copy session c1b68243a7144af5835b051aa9916d13-9d5700ab4d4f465bbe08ad25d0b9084f received from {username='slave'} at 127.0.99.135:33175
I20240501 07:58:46.727123 734 tablet_copy_service.cc:433] P e2c804beb8ef40ecb16c952f8875ca02: ending tablet copy session c1b68243a7144af5835b051aa9916d13-9d5700ab4d4f465bbe08ad25d0b9084f on tablet 9d5700ab4d4f465bbe08ad25d0b9084f with peer c1b68243a7144af5835b051aa9916d13
W20240501 07:58:46.744411 648 consensus_peers.cc:485] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 07:58:46.782863 2293 raft_consensus.cc:1203] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240501 07:58:46.865788 3403 tablet_copy_client.cc:523] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:46.867522 3403 tablet_bootstrap.cc:492] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750: Bootstrap starting.
I20240501 07:58:46.887076 3403 tablet_bootstrap.cc:492] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:46.887406 3403 tablet_bootstrap.cc:492] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750: Bootstrap complete.
I20240501 07:58:46.887585 3403 ts_tablet_manager.cc:1367] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.002s
I20240501 07:58:46.888010 3403 raft_consensus.cc:348] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } }
I20240501 07:58:46.888127 3403 raft_consensus.cc:729] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6645111fc9e446e8877750a3c4779750, State: Initialized, Role: LEARNER
I20240501 07:58:46.888208 3403 consensus_queue.cc:260] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } }
I20240501 07:58:46.888540 3403 ts_tablet_manager.cc:1398] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:46.889220 734 tablet_copy_service.cc:342] P e2c804beb8ef40ecb16c952f8875ca02: Request end of tablet copy session 6645111fc9e446e8877750a3c4779750-004e01dbd0f94cbeb20cba08164d386d received from {username='slave'} at 127.0.99.136:35421
I20240501 07:58:46.889310 734 tablet_copy_service.cc:433] P e2c804beb8ef40ecb16c952f8875ca02: ending tablet copy session 6645111fc9e446e8877750a3c4779750-004e01dbd0f94cbeb20cba08164d386d on tablet 004e01dbd0f94cbeb20cba08164d386d with peer 6645111fc9e446e8877750a3c4779750
I20240501 07:58:47.226056 2424 raft_consensus.cc:1203] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 07:58:47.375550 3514 raft_consensus.cc:1050] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02: attempting to promote NON_VOTER c1b68243a7144af5835b051aa9916d13 to VOTER
I20240501 07:58:47.376116 3514 consensus_queue.cc:237] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } }
I20240501 07:58:47.377727 975 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 07:58:47.377727 2424 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 07:58:47.378315 3515 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:58:47.378651 2293 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [term 2 LEARNER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 07:58:47.379108 3515 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:58:47.379292 3517 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:58:47.381002 3517 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, c1b68243a7144af5835b051aa9916d13 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:47.381513 975 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, c1b68243a7144af5835b051aa9916d13 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:47.382616 2293 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, c1b68243a7144af5835b051aa9916d13 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:47.383198 2424 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, c1b68243a7144af5835b051aa9916d13 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:47.384260 438 catalog_manager.cc:5484] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 reported cstate change: config changed from index 9 to 10, c1b68243a7144af5835b051aa9916d13 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:47.389112 714 consensus_queue.cc:237] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } }
I20240501 07:58:47.390496 2424 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 07:58:47.390564 2293 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [term 2 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 07:58:47.391001 3429 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 07:58:47.391364 3429 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 07:58:47.392689 3515 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:47.393080 2293 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:47.393443 2424 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:47.395128 426 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d5700ab4d4f465bbe08ad25d0b9084f with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:58:47.395315 433 catalog_manager.cc:5484] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: config changed from index 10 to 11, VOTER e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:47.397890 955 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d5700ab4d4f465bbe08ad25d0b9084f with delete_type TABLET_DATA_TOMBSTONED (TS e2edc87d834f4e0bb662d501af69f2b5 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:34188
I20240501 07:58:47.398396 3526 tablet_replica.cc:330] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5: stopping tablet replica
I20240501 07:58:47.398725 3526 raft_consensus.cc:2229] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:47.398962 3526 raft_consensus.cc:2258] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:47.400050 3526 ts_tablet_manager.cc:1875] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:47.401908 3526 ts_tablet_manager.cc:1888] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 07:58:47.402000 3526 log.cc:1196] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2edc87d834f4e0bb662d501af69f2b5: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/9d5700ab4d4f465bbe08ad25d0b9084f
I20240501 07:58:47.402588 427 catalog_manager.cc:4830] TS e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973): tablet 9d5700ab4d4f465bbe08ad25d0b9084f (table rebalance_test_table_2 [id=1f6bea7fb1b842af87d7bafed90d8fb6]) successfully deleted
W20240501 07:58:47.415655 910 consensus_peers.cc:485] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 07:58:47.594246 714 consensus_queue.cc:237] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } }
I20240501 07:58:47.595818 2424 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:58:47.595924 975 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 07:58:47.595960 649 consensus_peers.cc:485] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer c1b68243a7144af5835b051aa9916d13 (127.0.99.135:44501): Couldn't send request to peer c1b68243a7144af5835b051aa9916d13. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8e9192810694fa1bf54ca88fb619d11. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:47.596386 3514 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:58:47.596565 3517 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:58:47.598315 3429 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c1b68243a7144af5835b051aa9916d13 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } } }
I20240501 07:58:47.598801 975 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c1b68243a7144af5835b051aa9916d13 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } } }
I20240501 07:58:47.600224 2424 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c1b68243a7144af5835b051aa9916d13 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } } }
I20240501 07:58:47.601545 440 catalog_manager.cc:5484] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 reported cstate change: config changed from index 6 to 7, NON_VOTER c1b68243a7144af5835b051aa9916d13 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } } }
W20240501 07:58:47.688776 779 consensus_peers.cc:485] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 07:58:47.815696 3517 raft_consensus.cc:1050] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02: attempting to promote NON_VOTER 6645111fc9e446e8877750a3c4779750 to VOTER
I20240501 07:58:47.816174 3517 consensus_queue.cc:237] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } }
I20240501 07:58:47.817703 2162 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:58:47.817739 976 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:58:47.817919 2424 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750 [term 1 LEARNER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:58:47.818105 3517 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:58:47.818305 3429 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:58:47.818545 3514 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:58:47.820953 3429 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6645111fc9e446e8877750a3c4779750 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
I20240501 07:58:47.821413 2162 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6645111fc9e446e8877750a3c4779750 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
I20240501 07:58:47.821707 976 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6645111fc9e446e8877750a3c4779750 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
I20240501 07:58:47.823215 2424 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6645111fc9e446e8877750a3c4779750 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
I20240501 07:58:47.824028 432 catalog_manager.cc:5484] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: config changed from index 7 to 8, 6645111fc9e446e8877750a3c4779750 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:47.836122 714 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "004e01dbd0f94cbeb20cba08164d386d"
dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45744
I20240501 07:58:47.836416 714 raft_consensus.cc:595] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:48.055922 714 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "004e01dbd0f94cbeb20cba08164d386d"
dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45772
I20240501 07:58:48.056087 714 raft_consensus.cc:595] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:48.089809 3590 ts_tablet_manager.cc:902] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13: Initiating tablet copy from peer e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577)
I20240501 07:58:48.090317 3590 tablet_copy_client.cc:308] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43577
I20240501 07:58:48.090961 734 tablet_copy_service.cc:140] P e2c804beb8ef40ecb16c952f8875ca02: Received BeginTabletCopySession request for tablet b8e9192810694fa1bf54ca88fb619d11 from peer c1b68243a7144af5835b051aa9916d13 ({username='slave'} at 127.0.99.135:33175)
I20240501 07:58:48.091089 734 tablet_copy_service.cc:161] P e2c804beb8ef40ecb16c952f8875ca02: Beginning new tablet copy session on tablet b8e9192810694fa1bf54ca88fb619d11 from peer c1b68243a7144af5835b051aa9916d13 at {username='slave'} at 127.0.99.135:33175: session id = c1b68243a7144af5835b051aa9916d13-b8e9192810694fa1bf54ca88fb619d11
I20240501 07:58:48.091838 734 tablet_copy_source_session.cc:205] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:48.092862 3590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8e9192810694fa1bf54ca88fb619d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:48.094869 3590 tablet_copy_client.cc:783] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:48.095038 3590 tablet_copy_client.cc:647] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:48.096256 3594 tablet_copy_client.cc:1002] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13: tablet copy: Injecting latency into file download: 1500
W20240501 07:58:48.176720 648 consensus_peers.cc:485] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 07:58:48.248390 3429 raft_consensus.cc:979] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02: : Instructing follower 36272be7c71b413a94d36b4257d34425 to start an election
I20240501 07:58:48.248518 3429 raft_consensus.cc:1067] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Signalling peer 36272be7c71b413a94d36b4257d34425 to start an election
I20240501 07:58:48.249202 2162 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "004e01dbd0f94cbeb20cba08164d386d"
dest_uuid: "36272be7c71b413a94d36b4257d34425"
from {username='slave'} at 127.0.99.130:51417
I20240501 07:58:48.249395 2162 raft_consensus.cc:482] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:48.249473 2162 raft_consensus.cc:3037] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:48.250308 2162 raft_consensus.cc:504] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } }
I20240501 07:58:48.251422 2162 leader_election.cc:288] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [CANDIDATE]: Term 2 election: Requested vote from peers e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577), e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973), 6645111fc9e446e8877750a3c4779750 (127.0.99.136:44643)
I20240501 07:58:48.251757 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004e01dbd0f94cbeb20cba08164d386d" candidate_uuid: "36272be7c71b413a94d36b4257d34425" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02"
I20240501 07:58:48.251885 714 raft_consensus.cc:3032] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:48.251960 714 raft_consensus.cc:729] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Running, Role: LEADER
I20240501 07:58:48.252135 714 consensus_queue.cc:260] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } }
I20240501 07:58:48.252347 714 raft_consensus.cc:3037] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:48.253404 714 raft_consensus.cc:2454] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36272be7c71b413a94d36b4257d34425 in term 2.
I20240501 07:58:48.256405 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004e01dbd0f94cbeb20cba08164d386d" candidate_uuid: "36272be7c71b413a94d36b4257d34425" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "6645111fc9e446e8877750a3c4779750"
I20240501 07:58:48.256593 2424 raft_consensus.cc:3037] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:48.257397 2424 raft_consensus.cc:2454] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36272be7c71b413a94d36b4257d34425 in term 2.
I20240501 07:58:48.257645 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004e01dbd0f94cbeb20cba08164d386d" candidate_uuid: "36272be7c71b413a94d36b4257d34425" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e2edc87d834f4e0bb662d501af69f2b5"
I20240501 07:58:48.257787 976 raft_consensus.cc:3037] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:48.257792 2097 leader_election.cc:302] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 36272be7c71b413a94d36b4257d34425, 6645111fc9e446e8877750a3c4779750, e2c804beb8ef40ecb16c952f8875ca02; no voters:
I20240501 07:58:48.258304 3564 raft_consensus.cc:2781] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:48.258607 3564 raft_consensus.cc:686] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 2 LEADER]: Becoming Leader. State: Replica: 36272be7c71b413a94d36b4257d34425, State: Running, Role: LEADER
I20240501 07:58:48.258853 976 raft_consensus.cc:2454] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36272be7c71b413a94d36b4257d34425 in term 2.
I20240501 07:58:48.258987 3564 consensus_queue.cc:237] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } }
I20240501 07:58:48.261901 433 catalog_manager.cc:5484] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 reported cstate change: term changed from 1 to 2, leader changed from e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130) to 36272be7c71b413a94d36b4257d34425 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "36272be7c71b413a94d36b4257d34425" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:48.295464 779 consensus_peers.cc:485] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:58:48.373392 648 consensus_peers.cc:485] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:58:48.383024 910 consensus_peers.cc:485] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:58:48.415179 779 consensus_peers.cc:485] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:58:48.599144 426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240501 07:58:48.599994 426 catalog_manager.cc:4573] TS 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d5700ab4d4f465bbe08ad25d0b9084f: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
W20240501 07:58:48.633481 648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 07:58:48.680119 779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 07:58:48.722237 714 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 2 FOLLOWER]: Refusing update from remote peer 36272be7c71b413a94d36b4257d34425: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 07:58:48.722800 3564 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 07:58:48.743722 910 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 07:58:48.744279 910 consensus_peers.cc:485] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 07:58:48.785218 2424 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Refusing update from remote peer 36272be7c71b413a94d36b4257d34425: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 07:58:48.785719 3564 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:58:48.789119 976 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Refusing update from remote peer 36272be7c71b413a94d36b4257d34425: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 07:58:48.789673 3635 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 07:58:48.838467 910 consensus_peers.cc:485] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:58:48.902179 779 consensus_peers.cc:485] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:58:49.133095 648 consensus_peers.cc:485] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 07:58:49.228521 2162 consensus_queue.cc:237] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } }
I20240501 07:58:49.229921 976 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Refusing update from remote peer 36272be7c71b413a94d36b4257d34425: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 07:58:49.229964 2424 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Refusing update from remote peer 36272be7c71b413a94d36b4257d34425: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 07:58:49.230314 3637 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:58:49.230620 3564 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:58:49.232337 3637 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
I20240501 07:58:49.232787 976 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
I20240501 07:58:49.232780 2424 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } }
I20240501 07:58:49.234732 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 004e01dbd0f94cbeb20cba08164d386d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:58:49.235293 433 catalog_manager.cc:5484] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 reported cstate change: config changed from index 8 to 10, VOTER e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "36272be7c71b413a94d36b4257d34425" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:49.238035 692 tablet_service.cc:1514] Processing DeleteTablet for tablet 004e01dbd0f94cbeb20cba08164d386d with delete_type TABLET_DATA_TOMBSTONED (TS e2c804beb8ef40ecb16c952f8875ca02 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43240
I20240501 07:58:49.238557 3665 tablet_replica.cc:330] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02: stopping tablet replica
I20240501 07:58:49.238688 3665 raft_consensus.cc:2229] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:49.238806 3665 raft_consensus.cc:2258] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:49.239526 3665 ts_tablet_manager.cc:1875] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:49.241427 3665 ts_tablet_manager.cc:1888] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 07:58:49.241494 3665 log.cc:1196] T 004e01dbd0f94cbeb20cba08164d386d P e2c804beb8ef40ecb16c952f8875ca02: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/004e01dbd0f94cbeb20cba08164d386d
I20240501 07:58:49.242039 426 catalog_manager.cc:4830] TS e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577): tablet 004e01dbd0f94cbeb20cba08164d386d (table rebalance_test_table_3 [id=0678a1a47ecc4cb3bb664b77ec709a59]) successfully deleted
I20240501 07:58:49.596715 3590 tablet_copy_client.cc:523] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:49.608932 3590 tablet_bootstrap.cc:492] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13: Bootstrap starting.
I20240501 07:58:49.629938 3590 tablet_bootstrap.cc:492] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:49.630259 3590 tablet_bootstrap.cc:492] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13: Bootstrap complete.
I20240501 07:58:49.630463 3590 ts_tablet_manager.cc:1367] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13: Time spent bootstrapping tablet: real 0.022s user 0.021s sys 0.001s
I20240501 07:58:49.631106 3590 raft_consensus.cc:348] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } }
I20240501 07:58:49.631234 3590 raft_consensus.cc:729] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c1b68243a7144af5835b051aa9916d13, State: Initialized, Role: LEARNER
I20240501 07:58:49.631342 3590 consensus_queue.cc:260] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } }
I20240501 07:58:49.631716 3590 ts_tablet_manager.cc:1398] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:49.632369 734 tablet_copy_service.cc:342] P e2c804beb8ef40ecb16c952f8875ca02: Request end of tablet copy session c1b68243a7144af5835b051aa9916d13-b8e9192810694fa1bf54ca88fb619d11 received from {username='slave'} at 127.0.99.135:33175
I20240501 07:58:49.632475 734 tablet_copy_service.cc:433] P e2c804beb8ef40ecb16c952f8875ca02: ending tablet copy session c1b68243a7144af5835b051aa9916d13-b8e9192810694fa1bf54ca88fb619d11 on tablet b8e9192810694fa1bf54ca88fb619d11 with peer c1b68243a7144af5835b051aa9916d13
I20240501 07:58:49.636023 2293 raft_consensus.cc:1203] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240501 07:58:49.931861 910 consensus_peers.cc:485] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:58:50.157557 779 consensus_peers.cc:485] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 07:58:50.260138 3704 raft_consensus.cc:1050] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02: attempting to promote NON_VOTER c1b68243a7144af5835b051aa9916d13 to VOTER
I20240501 07:58:50.260643 3704 consensus_queue.cc:237] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } }
I20240501 07:58:50.262185 2293 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13 [term 1 LEARNER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:58:50.262257 976 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:58:50.262634 2424 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:58:50.262715 3706 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:58:50.262923 3623 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:58:50.263141 3704 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:58:50.265105 3623 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c1b68243a7144af5835b051aa9916d13 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:50.265784 976 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c1b68243a7144af5835b051aa9916d13 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:50.267342 2293 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c1b68243a7144af5835b051aa9916d13 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:50.268362 440 catalog_manager.cc:5484] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: config changed from index 7 to 8, c1b68243a7144af5835b051aa9916d13 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:50.269194 2424 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c1b68243a7144af5835b051aa9916d13 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:50.271775 714 consensus_queue.cc:237] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } }
I20240501 07:58:50.273553 2293 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 07:58:50.274147 2424 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 07:58:50.274356 3704 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:58:50.274576 3703 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:58:50.276764 3623 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:50.277444 2424 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:50.277455 2293 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:50.279018 426 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b8e9192810694fa1bf54ca88fb619d11 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:58:50.280263 433 catalog_manager.cc:5484] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13 reported cstate change: config changed from index 8 to 9, VOTER e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:50.282869 955 tablet_service.cc:1514] Processing DeleteTablet for tablet b8e9192810694fa1bf54ca88fb619d11 with delete_type TABLET_DATA_TOMBSTONED (TS e2edc87d834f4e0bb662d501af69f2b5 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34188
I20240501 07:58:50.283365 3716 tablet_replica.cc:330] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5: stopping tablet replica
I20240501 07:58:50.283484 3716 raft_consensus.cc:2229] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:50.283670 3716 raft_consensus.cc:2258] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:50.284258 3716 ts_tablet_manager.cc:1875] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:50.286080 3716 ts_tablet_manager.cc:1888] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 07:58:50.286164 3716 log.cc:1196] T b8e9192810694fa1bf54ca88fb619d11 P e2edc87d834f4e0bb662d501af69f2b5: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b8e9192810694fa1bf54ca88fb619d11
I20240501 07:58:50.286756 427 catalog_manager.cc:4830] TS e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973): tablet b8e9192810694fa1bf54ca88fb619d11 (table rebalance_test_table_1 [id=e08d5aefd3334c55a539f0f47df899e4]) successfully deleted
I20240501 07:58:50.442211 825 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:50.442518 2404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:50.451079 2011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:50.451646 2142 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:50.453603 692 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:50.458081 955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:50.462019 2273 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:50.504613 976 consensus_queue.cc:237] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } }
I20240501 07:58:50.506682 2014 raft_consensus.cc:1261] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:58:50.506716 843 raft_consensus.cc:1261] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:58:50.507557 3768 consensus_queue.cc:1035] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:58:50.507810 3767 consensus_queue.cc:1035] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:58:50.509884 3767 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 36272be7c71b413a94d36b4257d34425 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } } }
W20240501 07:58:50.510479 911 consensus_peers.cc:485] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 36272be7c71b413a94d36b4257d34425 (127.0.99.134:39945): Couldn't send request to peer 36272be7c71b413a94d36b4257d34425. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0638f250a164460a694cd41b3e9f27d. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:50.510514 2014 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 36272be7c71b413a94d36b4257d34425 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } } }
I20240501 07:58:50.510805 843 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 36272be7c71b413a94d36b4257d34425 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } } }
I20240501 07:58:50.513339 438 catalog_manager.cc:5484] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6 reported cstate change: config changed from index 6 to 7, NON_VOTER 36272be7c71b413a94d36b4257d34425 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "e2edc87d834f4e0bb662d501af69f2b5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } } }
I20240501 07:58:50.522183 2162 consensus_queue.cc:237] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } }
I20240501 07:58:50.524140 2424 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Refusing update from remote peer 36272be7c71b413a94d36b4257d34425: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 07:58:50.524329 976 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Refusing update from remote peer 36272be7c71b413a94d36b4257d34425: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 07:58:50.524538 3779 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 07:58:50.524724 3778 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 07:58:50.527772 3635 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER c1b68243a7144af5835b051aa9916d13 (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } } }
I20240501 07:58:50.528246 2424 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER c1b68243a7144af5835b051aa9916d13 (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } } }
I20240501 07:58:50.528295 976 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER c1b68243a7144af5835b051aa9916d13 (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } } }
W20240501 07:58:50.530273 2097 consensus_peers.cc:485] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 -> Peer c1b68243a7144af5835b051aa9916d13 (127.0.99.135:44501): Couldn't send request to peer c1b68243a7144af5835b051aa9916d13. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 004e01dbd0f94cbeb20cba08164d386d. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:50.531133 436 catalog_manager.cc:5484] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 reported cstate change: config changed from index 10 to 11, NON_VOTER c1b68243a7144af5835b051aa9916d13 (127.0.99.135) added. New cstate: current_term: 2 leader_uuid: "36272be7c71b413a94d36b4257d34425" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:50.540282 714 consensus_queue.cc:237] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: true } }
I20240501 07:58:50.542286 2424 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 07:58:50.542296 2293 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [term 2 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 07:58:50.542771 3704 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 07:58:50.543043 3703 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 07:58:50.545536 3623 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: true } } }
W20240501 07:58:50.545697 647 consensus_peers.cc:485] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 -> Peer b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133:39953): Couldn't send request to peer b331ee755f7d4df48eb9f4460a0340b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d5700ab4d4f465bbe08ad25d0b9084f. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:50.546073 2424 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: true } } }
I20240501 07:58:50.546097 2293 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: true } } }
I20240501 07:58:50.548818 432 catalog_manager.cc:5484] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 reported cstate change: config changed from index 11 to 12, NON_VOTER b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133) added. New cstate: current_term: 2 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: true } } }
W20240501 07:58:50.645936 648 consensus_peers.cc:485] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:58:50.742275 779 consensus_peers.cc:485] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 07:58:50.925154 3821 ts_tablet_manager.cc:902] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13: Initiating tablet copy from peer 36272be7c71b413a94d36b4257d34425 (127.0.99.134:39945)
I20240501 07:58:50.925772 3821 tablet_copy_client.cc:308] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:39945
I20240501 07:58:50.929478 2182 tablet_copy_service.cc:140] P 36272be7c71b413a94d36b4257d34425: Received BeginTabletCopySession request for tablet 004e01dbd0f94cbeb20cba08164d386d from peer c1b68243a7144af5835b051aa9916d13 ({username='slave'} at 127.0.99.135:48101)
I20240501 07:58:50.929705 2182 tablet_copy_service.cc:161] P 36272be7c71b413a94d36b4257d34425: Beginning new tablet copy session on tablet 004e01dbd0f94cbeb20cba08164d386d from peer c1b68243a7144af5835b051aa9916d13 at {username='slave'} at 127.0.99.135:48101: session id = c1b68243a7144af5835b051aa9916d13-004e01dbd0f94cbeb20cba08164d386d
I20240501 07:58:50.931166 2182 tablet_copy_source_session.cc:205] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:50.932466 3821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004e01dbd0f94cbeb20cba08164d386d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.934473 3821 tablet_copy_client.cc:783] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:50.934628 3821 tablet_copy_client.cc:647] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:50.936029 3827 ts_tablet_manager.cc:902] T e0638f250a164460a694cd41b3e9f27d P 36272be7c71b413a94d36b4257d34425: Initiating tablet copy from peer e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973)
I20240501 07:58:50.936205 3825 tablet_copy_client.cc:1002] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13: tablet copy: Injecting latency into file download: 1500
I20240501 07:58:50.936573 3827 tablet_copy_client.cc:308] T e0638f250a164460a694cd41b3e9f27d P 36272be7c71b413a94d36b4257d34425: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36973
I20240501 07:58:50.937110 996 tablet_copy_service.cc:140] P e2edc87d834f4e0bb662d501af69f2b5: Received BeginTabletCopySession request for tablet e0638f250a164460a694cd41b3e9f27d from peer 36272be7c71b413a94d36b4257d34425 ({username='slave'} at 127.0.99.134:43167)
I20240501 07:58:50.937232 996 tablet_copy_service.cc:161] P e2edc87d834f4e0bb662d501af69f2b5: Beginning new tablet copy session on tablet e0638f250a164460a694cd41b3e9f27d from peer 36272be7c71b413a94d36b4257d34425 at {username='slave'} at 127.0.99.134:43167: session id = 36272be7c71b413a94d36b4257d34425-e0638f250a164460a694cd41b3e9f27d
I20240501 07:58:50.938148 996 tablet_copy_source_session.cc:205] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:50.939150 3827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0638f250a164460a694cd41b3e9f27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:50.940999 3827 tablet_copy_client.cc:783] T e0638f250a164460a694cd41b3e9f27d P 36272be7c71b413a94d36b4257d34425: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:50.941170 3827 tablet_copy_client.cc:647] T e0638f250a164460a694cd41b3e9f27d P 36272be7c71b413a94d36b4257d34425: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:50.942490 3831 tablet_copy_client.cc:1002] T e0638f250a164460a694cd41b3e9f27d P 36272be7c71b413a94d36b4257d34425: tablet copy: Injecting latency into file download: 1500
W20240501 07:58:51.059768 648 consensus_peers.cc:485] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:58:51.064532 910 consensus_peers.cc:485] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:58:51.142047 779 consensus_peers.cc:485] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 07:58:51.144194 3847 ts_tablet_manager.cc:902] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6: Initiating tablet copy from peer e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577)
I20240501 07:58:51.144697 3847 tablet_copy_client.cc:308] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43577
I20240501 07:58:51.148787 734 tablet_copy_service.cc:140] P e2c804beb8ef40ecb16c952f8875ca02: Received BeginTabletCopySession request for tablet 9d5700ab4d4f465bbe08ad25d0b9084f from peer b331ee755f7d4df48eb9f4460a0340b6 ({username='slave'} at 127.0.99.133:50035)
I20240501 07:58:51.148938 734 tablet_copy_service.cc:161] P e2c804beb8ef40ecb16c952f8875ca02: Beginning new tablet copy session on tablet 9d5700ab4d4f465bbe08ad25d0b9084f from peer b331ee755f7d4df48eb9f4460a0340b6 at {username='slave'} at 127.0.99.133:50035: session id = b331ee755f7d4df48eb9f4460a0340b6-9d5700ab4d4f465bbe08ad25d0b9084f
I20240501 07:58:51.149711 734 tablet_copy_source_session.cc:205] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:51.150777 3847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d5700ab4d4f465bbe08ad25d0b9084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:51.152830 3847 tablet_copy_client.cc:783] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:51.153023 3847 tablet_copy_client.cc:647] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:51.155088 3851 tablet_copy_client.cc:1002] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6: tablet copy: Injecting latency into file download: 1500
W20240501 07:58:51.220855 910 consensus_peers.cc:485] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:58:51.255499 910 consensus_peers.cc:485] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:58:51.355345 779 consensus_peers.cc:485] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:58:51.798061 648 consensus_peers.cc:485] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:58:52.408344 910 consensus_peers.cc:485] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 07:58:52.436566 3821 tablet_copy_client.cc:523] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:52.437924 3821 tablet_bootstrap.cc:492] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13: Bootstrap starting.
I20240501 07:58:52.442909 3827 tablet_copy_client.cc:523] T e0638f250a164460a694cd41b3e9f27d P 36272be7c71b413a94d36b4257d34425: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:52.444444 3827 tablet_bootstrap.cc:492] T e0638f250a164460a694cd41b3e9f27d P 36272be7c71b413a94d36b4257d34425: Bootstrap starting.
I20240501 07:58:52.466059 3821 tablet_bootstrap.cc:492] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:52.466493 3821 tablet_bootstrap.cc:492] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13: Bootstrap complete.
I20240501 07:58:52.466743 3821 ts_tablet_manager.cc:1367] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13: Time spent bootstrapping tablet: real 0.029s user 0.029s sys 0.001s
I20240501 07:58:52.467245 3821 raft_consensus.cc:348] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } }
I20240501 07:58:52.467501 3821 raft_consensus.cc:729] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c1b68243a7144af5835b051aa9916d13, State: Initialized, Role: LEARNER
I20240501 07:58:52.467635 3821 consensus_queue.cc:260] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: true } }
I20240501 07:58:52.468081 3821 ts_tablet_manager.cc:1398] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:52.468797 2182 tablet_copy_service.cc:342] P 36272be7c71b413a94d36b4257d34425: Request end of tablet copy session c1b68243a7144af5835b051aa9916d13-004e01dbd0f94cbeb20cba08164d386d received from {username='slave'} at 127.0.99.135:48101
I20240501 07:58:52.468910 2182 tablet_copy_service.cc:433] P 36272be7c71b413a94d36b4257d34425: ending tablet copy session c1b68243a7144af5835b051aa9916d13-004e01dbd0f94cbeb20cba08164d386d on tablet 004e01dbd0f94cbeb20cba08164d386d with peer c1b68243a7144af5835b051aa9916d13
I20240501 07:58:52.472920 3827 tablet_bootstrap.cc:492] T e0638f250a164460a694cd41b3e9f27d P 36272be7c71b413a94d36b4257d34425: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:52.473217 3827 tablet_bootstrap.cc:492] T e0638f250a164460a694cd41b3e9f27d P 36272be7c71b413a94d36b4257d34425: Bootstrap complete.
I20240501 07:58:52.473387 3827 ts_tablet_manager.cc:1367] T e0638f250a164460a694cd41b3e9f27d P 36272be7c71b413a94d36b4257d34425: Time spent bootstrapping tablet: real 0.029s user 0.026s sys 0.000s
I20240501 07:58:52.473891 3827 raft_consensus.cc:348] T e0638f250a164460a694cd41b3e9f27d P 36272be7c71b413a94d36b4257d34425 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } }
I20240501 07:58:52.474004 3827 raft_consensus.cc:729] T e0638f250a164460a694cd41b3e9f27d P 36272be7c71b413a94d36b4257d34425 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 36272be7c71b413a94d36b4257d34425, State: Initialized, Role: LEARNER
I20240501 07:58:52.474095 3827 consensus_queue.cc:260] T e0638f250a164460a694cd41b3e9f27d P 36272be7c71b413a94d36b4257d34425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } }
I20240501 07:58:52.474344 3827 ts_tablet_manager.cc:1398] T e0638f250a164460a694cd41b3e9f27d P 36272be7c71b413a94d36b4257d34425: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:52.474985 996 tablet_copy_service.cc:342] P e2edc87d834f4e0bb662d501af69f2b5: Request end of tablet copy session 36272be7c71b413a94d36b4257d34425-e0638f250a164460a694cd41b3e9f27d received from {username='slave'} at 127.0.99.134:43167
I20240501 07:58:52.475126 996 tablet_copy_service.cc:433] P e2edc87d834f4e0bb662d501af69f2b5: ending tablet copy session 36272be7c71b413a94d36b4257d34425-e0638f250a164460a694cd41b3e9f27d on tablet e0638f250a164460a694cd41b3e9f27d with peer 36272be7c71b413a94d36b4257d34425
I20240501 07:58:52.486716 2162 raft_consensus.cc:1203] T e0638f250a164460a694cd41b3e9f27d P 36272be7c71b413a94d36b4257d34425 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 07:58:52.655443 3847 tablet_copy_client.cc:523] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:52.656919 3847 tablet_bootstrap.cc:492] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6: Bootstrap starting.
I20240501 07:58:52.678016 3847 tablet_bootstrap.cc:492] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:52.678331 3847 tablet_bootstrap.cc:492] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6: Bootstrap complete.
I20240501 07:58:52.678560 3847 ts_tablet_manager.cc:1367] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6: Time spent bootstrapping tablet: real 0.022s user 0.023s sys 0.000s
I20240501 07:58:52.679164 3847 raft_consensus.cc:348] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: true } }
I20240501 07:58:52.679297 3847 raft_consensus.cc:729] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b331ee755f7d4df48eb9f4460a0340b6, State: Initialized, Role: LEARNER
I20240501 07:58:52.679404 3847 consensus_queue.cc:260] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: true } }
I20240501 07:58:52.679783 3847 ts_tablet_manager.cc:1398] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:52.680550 734 tablet_copy_service.cc:342] P e2c804beb8ef40ecb16c952f8875ca02: Request end of tablet copy session b331ee755f7d4df48eb9f4460a0340b6-9d5700ab4d4f465bbe08ad25d0b9084f received from {username='slave'} at 127.0.99.133:50035
I20240501 07:58:52.680647 734 tablet_copy_service.cc:433] P e2c804beb8ef40ecb16c952f8875ca02: ending tablet copy session b331ee755f7d4df48eb9f4460a0340b6-9d5700ab4d4f465bbe08ad25d0b9084f on tablet 9d5700ab4d4f465bbe08ad25d0b9084f with peer b331ee755f7d4df48eb9f4460a0340b6
I20240501 07:58:52.682850 2014 raft_consensus.cc:1203] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
W20240501 07:58:52.808586 779 consensus_peers.cc:485] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 07:58:52.916878 3989 raft_consensus.cc:1050] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5: attempting to promote NON_VOTER 36272be7c71b413a94d36b4257d34425 to VOTER
I20240501 07:58:52.917285 3989 consensus_queue.cc:237] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } }
I20240501 07:58:52.918659 2014 raft_consensus.cc:1261] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:58:52.918680 2162 raft_consensus.cc:1261] T e0638f250a164460a694cd41b3e9f27d P 36272be7c71b413a94d36b4257d34425 [term 1 LEARNER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:58:52.918716 843 raft_consensus.cc:1261] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:58:52.919090 3887 consensus_queue.cc:1035] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:58:52.919297 3956 consensus_queue.cc:1035] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:58:52.919471 3991 consensus_queue.cc:1035] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:58:52.921387 3991 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 36272be7c71b413a94d36b4257d34425 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:52.921896 2162 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P 36272be7c71b413a94d36b4257d34425 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 36272be7c71b413a94d36b4257d34425 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:52.921958 843 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 36272be7c71b413a94d36b4257d34425 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:52.924221 2014 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 36272be7c71b413a94d36b4257d34425 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:52.924510 436 catalog_manager.cc:5484] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 reported cstate change: config changed from index 7 to 8, 36272be7c71b413a94d36b4257d34425 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e2edc87d834f4e0bb662d501af69f2b5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } attrs { replace: true } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:52.930003 976 consensus_queue.cc:237] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } }
I20240501 07:58:52.931306 2014 raft_consensus.cc:1261] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 07:58:52.931736 3956 consensus_queue.cc:1035] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:58:52.931777 2162 raft_consensus.cc:1261] T e0638f250a164460a694cd41b3e9f27d P 36272be7c71b413a94d36b4257d34425 [term 1 FOLLOWER]: Refusing update from remote peer e2edc87d834f4e0bb662d501af69f2b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 07:58:52.932188 3956 consensus_queue.cc:1035] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:58:52.933732 3991 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 10aab83192f447829c9b8254e36d94b4 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:52.934180 2014 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P b331ee755f7d4df48eb9f4460a0340b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 10aab83192f447829c9b8254e36d94b4 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:52.934393 2162 raft_consensus.cc:2932] T e0638f250a164460a694cd41b3e9f27d P 36272be7c71b413a94d36b4257d34425 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 10aab83192f447829c9b8254e36d94b4 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:52.936203 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e0638f250a164460a694cd41b3e9f27d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:58:52.936419 433 catalog_manager.cc:5484] T e0638f250a164460a694cd41b3e9f27d P e2edc87d834f4e0bb662d501af69f2b5 reported cstate change: config changed from index 8 to 9, VOTER 10aab83192f447829c9b8254e36d94b4 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "e2edc87d834f4e0bb662d501af69f2b5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:52.938905 825 tablet_service.cc:1514] Processing DeleteTablet for tablet e0638f250a164460a694cd41b3e9f27d with delete_type TABLET_DATA_TOMBSTONED (TS 10aab83192f447829c9b8254e36d94b4 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:50386
I20240501 07:58:52.939286 3999 tablet_replica.cc:330] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4: stopping tablet replica
I20240501 07:58:52.939432 3999 raft_consensus.cc:2229] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:52.939656 3999 raft_consensus.cc:2258] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:52.940327 3999 ts_tablet_manager.cc:1875] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:52.942274 3999 ts_tablet_manager.cc:1888] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 07:58:52.942364 3999 log.cc:1196] T e0638f250a164460a694cd41b3e9f27d P 10aab83192f447829c9b8254e36d94b4: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e0638f250a164460a694cd41b3e9f27d
I20240501 07:58:52.943079 425 catalog_manager.cc:4830] TS 10aab83192f447829c9b8254e36d94b4 (127.0.99.131:40475): tablet e0638f250a164460a694cd41b3e9f27d (table rebalance_test_table_1 [id=e08d5aefd3334c55a539f0f47df899e4]) successfully deleted
I20240501 07:58:52.948635 2293 raft_consensus.cc:1203] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20240501 07:58:53.189816 4025 raft_consensus.cc:1050] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02: attempting to promote NON_VOTER b331ee755f7d4df48eb9f4460a0340b6 to VOTER
I20240501 07:58:53.190223 4025 consensus_queue.cc:237] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } }
I20240501 07:58:53.191556 2293 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [term 2 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 07:58:53.191555 2014 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6 [term 2 LEARNER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 07:58:53.191987 4026 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 07:58:53.192199 2424 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 07:58:53.192194 4025 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 07:58:53.192539 4026 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 07:58:53.193995 4025 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } }
I20240501 07:58:53.194448 2293 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } }
I20240501 07:58:53.195346 2424 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } }
I20240501 07:58:53.196822 438 catalog_manager.cc:5484] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: config changed from index 12 to 13, b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:53.197202 2014 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } }
I20240501 07:58:53.221002 714 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9d5700ab4d4f465bbe08ad25d0b9084f"
dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42342
I20240501 07:58:53.221168 714 raft_consensus.cc:595] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 2 LEADER]: Received request to transfer leadership
W20240501 07:58:53.235903 648 consensus_peers.cc:485] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:58:53.237828 779 consensus_peers.cc:485] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:58:53.314486 648 consensus_peers.cc:485] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 07:58:53.439664 714 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9d5700ab4d4f465bbe08ad25d0b9084f"
dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42350
I20240501 07:58:53.439846 714 raft_consensus.cc:595] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 2 LEADER]: Received request to transfer leadership
I20240501 07:58:53.567112 3905 raft_consensus.cc:1050] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425: attempting to promote NON_VOTER c1b68243a7144af5835b051aa9916d13 to VOTER
I20240501 07:58:53.567589 3905 consensus_queue.cc:237] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } }
I20240501 07:58:53.569010 2293 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13 [term 2 LEARNER]: Refusing update from remote peer 36272be7c71b413a94d36b4257d34425: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 07:58:53.569010 2424 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Refusing update from remote peer 36272be7c71b413a94d36b4257d34425: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 07:58:53.569103 976 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Refusing update from remote peer 36272be7c71b413a94d36b4257d34425: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 07:58:53.569475 4066 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 07:58:53.569706 4063 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 07:58:53.569844 4065 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 07:58:53.572736 4063 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, c1b68243a7144af5835b051aa9916d13 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:53.573202 976 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, c1b68243a7144af5835b051aa9916d13 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:53.573235 2293 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, c1b68243a7144af5835b051aa9916d13 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:53.575131 2424 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, c1b68243a7144af5835b051aa9916d13 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:53.575559 432 catalog_manager.cc:5484] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 reported cstate change: config changed from index 11 to 12, c1b68243a7144af5835b051aa9916d13 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "36272be7c71b413a94d36b4257d34425" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:53.578999 2162 consensus_queue.cc:237] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } }
I20240501 07:58:53.580396 2293 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13 [term 2 FOLLOWER]: Refusing update from remote peer 36272be7c71b413a94d36b4257d34425: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 07:58:53.580411 2424 raft_consensus.cc:1261] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Refusing update from remote peer 36272be7c71b413a94d36b4257d34425: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 07:58:53.580776 4066 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 07:58:53.580976 3905 consensus_queue.cc:1035] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 07:58:53.582756 4066 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P 36272be7c71b413a94d36b4257d34425 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:53.583120 2424 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:53.583148 2293 raft_consensus.cc:2932] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:53.584527 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 004e01dbd0f94cbeb20cba08164d386d with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:58:53.585637 438 catalog_manager.cc:5484] T 004e01dbd0f94cbeb20cba08164d386d P c1b68243a7144af5835b051aa9916d13 reported cstate change: config changed from index 12 to 13, VOTER e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "36272be7c71b413a94d36b4257d34425" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } }
I20240501 07:58:53.587070 4026 raft_consensus.cc:979] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02: : Instructing follower c1b68243a7144af5835b051aa9916d13 to start an election
I20240501 07:58:53.587170 4026 raft_consensus.cc:1067] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 2 LEADER]: Signalling peer c1b68243a7144af5835b051aa9916d13 to start an election
I20240501 07:58:53.587579 2293 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d5700ab4d4f465bbe08ad25d0b9084f"
dest_uuid: "c1b68243a7144af5835b051aa9916d13"
from {username='slave'} at 127.0.99.130:51733
I20240501 07:58:53.587757 2293 raft_consensus.cc:482] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:53.587855 2293 raft_consensus.cc:3037] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:58:53.588413 955 tablet_service.cc:1514] Processing DeleteTablet for tablet 004e01dbd0f94cbeb20cba08164d386d with delete_type TABLET_DATA_TOMBSTONED (TS e2edc87d834f4e0bb662d501af69f2b5 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:34188
I20240501 07:58:53.588800 4070 tablet_replica.cc:330] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5: stopping tablet replica
I20240501 07:58:53.588851 2293 raft_consensus.cc:504] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } }
I20240501 07:58:53.589057 4070 raft_consensus.cc:2229] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:53.589356 4070 raft_consensus.cc:2258] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:53.590109 4070 ts_tablet_manager.cc:1875] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:53.590411 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5700ab4d4f465bbe08ad25d0b9084f" candidate_uuid: "c1b68243a7144af5835b051aa9916d13" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02"
I20240501 07:58:53.590586 714 raft_consensus.cc:3032] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:58:53.590636 714 raft_consensus.cc:729] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Running, Role: LEADER
I20240501 07:58:53.590790 714 consensus_queue.cc:260] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } }
I20240501 07:58:53.590975 714 raft_consensus.cc:3037] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:58:53.591078 2293 leader_election.cc:288] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [CANDIDATE]: Term 3 election: Requested vote from peers e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577), 6645111fc9e446e8877750a3c4779750 (127.0.99.136:44643), b331ee755f7d4df48eb9f4460a0340b6 (127.0.99.133:39953)
I20240501 07:58:53.591919 714 raft_consensus.cc:2454] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1b68243a7144af5835b051aa9916d13 in term 3.
I20240501 07:58:53.592944 4070 ts_tablet_manager.cc:1888] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240501 07:58:53.593029 4070 log.cc:1196] T 004e01dbd0f94cbeb20cba08164d386d P e2edc87d834f4e0bb662d501af69f2b5: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/004e01dbd0f94cbeb20cba08164d386d
I20240501 07:58:53.593596 427 catalog_manager.cc:4830] TS e2edc87d834f4e0bb662d501af69f2b5 (127.0.99.132:36973): tablet 004e01dbd0f94cbeb20cba08164d386d (table rebalance_test_table_3 [id=0678a1a47ecc4cb3bb664b77ec709a59]) successfully deleted
I20240501 07:58:53.595878 2014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5700ab4d4f465bbe08ad25d0b9084f" candidate_uuid: "c1b68243a7144af5835b051aa9916d13" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "b331ee755f7d4df48eb9f4460a0340b6"
I20240501 07:58:53.596112 2014 raft_consensus.cc:3037] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:58:53.596539 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d5700ab4d4f465bbe08ad25d0b9084f" candidate_uuid: "c1b68243a7144af5835b051aa9916d13" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "6645111fc9e446e8877750a3c4779750"
I20240501 07:58:53.596680 2424 raft_consensus.cc:3037] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:58:53.597245 2014 raft_consensus.cc:2454] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1b68243a7144af5835b051aa9916d13 in term 3.
I20240501 07:58:53.597556 2424 raft_consensus.cc:2454] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1b68243a7144af5835b051aa9916d13 in term 3.
I20240501 07:58:53.597628 2226 leader_election.cc:302] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: b331ee755f7d4df48eb9f4460a0340b6, c1b68243a7144af5835b051aa9916d13, e2c804beb8ef40ecb16c952f8875ca02; no voters:
I20240501 07:58:53.597911 3955 raft_consensus.cc:2781] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:58:53.598122 3955 raft_consensus.cc:686] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [term 3 LEADER]: Becoming Leader. State: Replica: c1b68243a7144af5835b051aa9916d13, State: Running, Role: LEADER
I20240501 07:58:53.598404 3955 consensus_queue.cc:237] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } }
I20240501 07:58:53.601050 436 catalog_manager.cc:5484] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 reported cstate change: term changed from 2 to 3, leader changed from e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130) to c1b68243a7144af5835b051aa9916d13 (127.0.99.135). New cstate: current_term: 3 leader_uuid: "c1b68243a7144af5835b051aa9916d13" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:53.611863 910 consensus_peers.cc:485] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:58:53.702409 648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 07:58:53.747339 910 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 07:58:53.748025 910 consensus_peers.cc:485] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:58:53.794966 779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 07:58:53.795749 779 consensus_peers.cc:485] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:58:53.918511 779 consensus_peers.cc:485] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:58:53.967609 910 consensus_peers.cc:485] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 07:58:53.983697 2424 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 3 FOLLOWER]: Refusing update from remote peer c1b68243a7144af5835b051aa9916d13: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 07:58:53.984223 3955 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 07:58:54.073812 2014 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6 [term 3 FOLLOWER]: Refusing update from remote peer c1b68243a7144af5835b051aa9916d13: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 07:58:54.074306 3955 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 07:58:54.077494 714 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 3 FOLLOWER]: Refusing update from remote peer c1b68243a7144af5835b051aa9916d13: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 07:58:54.078184 4099 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 07:58:54.085551 2293 consensus_queue.cc:237] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } }
I20240501 07:58:54.086839 2014 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6 [term 3 FOLLOWER]: Refusing update from remote peer c1b68243a7144af5835b051aa9916d13: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240501 07:58:54.087206 2424 raft_consensus.cc:1261] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 3 FOLLOWER]: Refusing update from remote peer c1b68243a7144af5835b051aa9916d13: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240501 07:58:54.087301 4099 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 07:58:54.087668 4075 consensus_queue.cc:1035] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 07:58:54.088913 4099 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } }
I20240501 07:58:54.089372 2014 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P b331ee755f7d4df48eb9f4460a0340b6 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } }
I20240501 07:58:54.089673 2424 raft_consensus.cc:2932] T 9d5700ab4d4f465bbe08ad25d0b9084f P 6645111fc9e446e8877750a3c4779750 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } } }
I20240501 07:58:54.091022 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d5700ab4d4f465bbe08ad25d0b9084f with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:58:54.092020 436 catalog_manager.cc:5484] T 9d5700ab4d4f465bbe08ad25d0b9084f P c1b68243a7144af5835b051aa9916d13 reported cstate change: config changed from index 13 to 15, VOTER e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130) evicted. New cstate: current_term: 3 leader_uuid: "c1b68243a7144af5835b051aa9916d13" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b331ee755f7d4df48eb9f4460a0340b6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39953 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:54.094388 692 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d5700ab4d4f465bbe08ad25d0b9084f with delete_type TABLET_DATA_TOMBSTONED (TS e2c804beb8ef40ecb16c952f8875ca02 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:43240
I20240501 07:58:54.094789 4110 tablet_replica.cc:330] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02: stopping tablet replica
I20240501 07:58:54.094908 4110 raft_consensus.cc:2229] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:54.095000 4110 raft_consensus.cc:2258] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:54.095538 4110 ts_tablet_manager.cc:1875] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:54.097344 4110 ts_tablet_manager.cc:1888] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20240501 07:58:54.097409 4110 log.cc:1196] T 9d5700ab4d4f465bbe08ad25d0b9084f P e2c804beb8ef40ecb16c952f8875ca02: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/9d5700ab4d4f465bbe08ad25d0b9084f
I20240501 07:58:54.097877 426 catalog_manager.cc:4830] TS e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577): tablet 9d5700ab4d4f465bbe08ad25d0b9084f (table rebalance_test_table_2 [id=1f6bea7fb1b842af87d7bafed90d8fb6]) successfully deleted
I20240501 07:58:54.300067 714 consensus_queue.cc:237] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } }
I20240501 07:58:54.301796 2293 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 07:58:54.301890 649 consensus_peers.cc:485] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 36272be7c71b413a94d36b4257d34425 (127.0.99.134:39945): Couldn't send request to peer 36272be7c71b413a94d36b4257d34425. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8e9192810694fa1bf54ca88fb619d11. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:54.301914 2424 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 07:58:54.302569 4124 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:58:54.302814 4123 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:58:54.304778 3932 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 36272be7c71b413a94d36b4257d34425 (127.0.99.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } } }
I20240501 07:58:54.305372 2293 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 36272be7c71b413a94d36b4257d34425 (127.0.99.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } } }
I20240501 07:58:54.306576 2424 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 36272be7c71b413a94d36b4257d34425 (127.0.99.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } } }
I20240501 07:58:54.308837 433 catalog_manager.cc:5484] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13 reported cstate change: config changed from index 9 to 10, NON_VOTER 36272be7c71b413a94d36b4257d34425 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } } }
W20240501 07:58:54.334111 648 consensus_peers.cc:485] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 07:58:54.709285 4137 ts_tablet_manager.cc:902] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425: Initiating tablet copy from peer e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577)
I20240501 07:58:54.709939 4137 tablet_copy_client.cc:308] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43577
I20240501 07:58:54.710574 734 tablet_copy_service.cc:140] P e2c804beb8ef40ecb16c952f8875ca02: Received BeginTabletCopySession request for tablet b8e9192810694fa1bf54ca88fb619d11 from peer 36272be7c71b413a94d36b4257d34425 ({username='slave'} at 127.0.99.134:53971)
I20240501 07:58:54.710703 734 tablet_copy_service.cc:161] P e2c804beb8ef40ecb16c952f8875ca02: Beginning new tablet copy session on tablet b8e9192810694fa1bf54ca88fb619d11 from peer 36272be7c71b413a94d36b4257d34425 at {username='slave'} at 127.0.99.134:53971: session id = 36272be7c71b413a94d36b4257d34425-b8e9192810694fa1bf54ca88fb619d11
I20240501 07:58:54.711522 734 tablet_copy_source_session.cc:205] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:54.712529 4137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8e9192810694fa1bf54ca88fb619d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.714757 4137 tablet_copy_client.cc:783] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:54.714926 4137 tablet_copy_client.cc:647] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:54.716043 4140 tablet_copy_client.cc:1002] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425: tablet copy: Injecting latency into file download: 1500
W20240501 07:58:54.827853 910 consensus_peers.cc:485] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:58:55.360256 779 consensus_peers.cc:485] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:58:55.444643 648 consensus_peers.cc:485] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:58:55.696143 648 consensus_peers.cc:485] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:58:55.799696 779 consensus_peers.cc:485] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:58:55.873440 910 consensus_peers.cc:485] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 07:58:56.216431 4137 tablet_copy_client.cc:523] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:56.217839 4137 tablet_bootstrap.cc:492] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425: Bootstrap starting.
I20240501 07:58:56.237447 4137 tablet_bootstrap.cc:492] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:56.237754 4137 tablet_bootstrap.cc:492] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425: Bootstrap complete.
I20240501 07:58:56.237915 4137 ts_tablet_manager.cc:1367] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20240501 07:58:56.238324 4137 raft_consensus.cc:348] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } }
I20240501 07:58:56.238493 4137 raft_consensus.cc:729] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 36272be7c71b413a94d36b4257d34425, State: Initialized, Role: LEARNER
I20240501 07:58:56.238632 4137 consensus_queue.cc:260] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: true } }
I20240501 07:58:56.238930 4137 ts_tablet_manager.cc:1398] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240501 07:58:56.239574 734 tablet_copy_service.cc:342] P e2c804beb8ef40ecb16c952f8875ca02: Request end of tablet copy session 36272be7c71b413a94d36b4257d34425-b8e9192810694fa1bf54ca88fb619d11 received from {username='slave'} at 127.0.99.134:53971
I20240501 07:58:56.239651 734 tablet_copy_service.cc:433] P e2c804beb8ef40ecb16c952f8875ca02: ending tablet copy session 36272be7c71b413a94d36b4257d34425-b8e9192810694fa1bf54ca88fb619d11 on tablet b8e9192810694fa1bf54ca88fb619d11 with peer 36272be7c71b413a94d36b4257d34425
W20240501 07:58:56.414860 779 consensus_peers.cc:485] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:58:56.459022 910 consensus_peers.cc:485] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 07:58:56.617398 2162 raft_consensus.cc:1203] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
W20240501 07:58:56.648919 779 consensus_peers.cc:485] T 88887f8c213f46b2b4d4555d6a13c346 P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:58:56.712087 910 consensus_peers.cc:485] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:58:56.805644 426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
W20240501 07:58:56.806465 426 catalog_manager.cc:4573] TS 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d5700ab4d4f465bbe08ad25d0b9084f: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111)
I20240501 07:58:57.099723 4158 raft_consensus.cc:1050] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02: attempting to promote NON_VOTER 36272be7c71b413a94d36b4257d34425 to VOTER
I20240501 07:58:57.100303 4158 consensus_queue.cc:237] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } }
I20240501 07:58:57.101696 2293 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 07:58:57.101742 2424 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 1 FOLLOWER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 07:58:57.102058 2162 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [term 1 LEARNER]: Refusing update from remote peer e2c804beb8ef40ecb16c952f8875ca02: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 07:58:57.102077 4213 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 07:58:57.102367 4158 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 07:58:57.102715 4158 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 07:58:57.104805 4158 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 36272be7c71b413a94d36b4257d34425 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:57.105293 2162 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 36272be7c71b413a94d36b4257d34425 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:57.107300 2293 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 36272be7c71b413a94d36b4257d34425 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:57.107735 438 catalog_manager.cc:5484] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 reported cstate change: config changed from index 10 to 11, 36272be7c71b413a94d36b4257d34425 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e2c804beb8ef40ecb16c952f8875ca02" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:57.108649 2424 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 36272be7c71b413a94d36b4257d34425 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:57.214751 714 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b8e9192810694fa1bf54ca88fb619d11"
dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42512
I20240501 07:58:57.214910 714 raft_consensus.cc:595] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Received request to transfer leadership
W20240501 07:58:57.260390 648 consensus_peers.cc:485] T e7eefec740384c688dc6e149b5ea2769 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 07:58:57.427117 714 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b8e9192810694fa1bf54ca88fb619d11"
dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42536
I20240501 07:58:57.427381 714 raft_consensus.cc:595] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:57.507522 4218 raft_consensus.cc:979] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02: : Instructing follower 36272be7c71b413a94d36b4257d34425 to start an election
I20240501 07:58:57.507642 4218 raft_consensus.cc:1067] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Signalling peer 36272be7c71b413a94d36b4257d34425 to start an election
I20240501 07:58:57.508155 2162 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8e9192810694fa1bf54ca88fb619d11"
dest_uuid: "36272be7c71b413a94d36b4257d34425"
from {username='slave'} at 127.0.99.130:51417
I20240501 07:58:57.508296 2162 raft_consensus.cc:482] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:57.508359 2162 raft_consensus.cc:3037] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:57.509181 2162 raft_consensus.cc:504] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } }
I20240501 07:58:57.509600 2162 leader_election.cc:288] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [CANDIDATE]: Term 2 election: Requested vote from peers e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577), 6645111fc9e446e8877750a3c4779750 (127.0.99.136:44643), c1b68243a7144af5835b051aa9916d13 (127.0.99.135:44501)
I20240501 07:58:57.509985 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e9192810694fa1bf54ca88fb619d11" candidate_uuid: "36272be7c71b413a94d36b4257d34425" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "e2c804beb8ef40ecb16c952f8875ca02"
I20240501 07:58:57.510058 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e9192810694fa1bf54ca88fb619d11" candidate_uuid: "36272be7c71b413a94d36b4257d34425" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "6645111fc9e446e8877750a3c4779750"
I20240501 07:58:57.510118 714 raft_consensus.cc:3032] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:57.510156 714 raft_consensus.cc:729] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2c804beb8ef40ecb16c952f8875ca02, State: Running, Role: LEADER
I20240501 07:58:57.510093 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8e9192810694fa1bf54ca88fb619d11" candidate_uuid: "36272be7c71b413a94d36b4257d34425" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "c1b68243a7144af5835b051aa9916d13"
I20240501 07:58:57.510195 2424 raft_consensus.cc:3037] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:57.510267 2293 raft_consensus.cc:3037] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:57.510314 714 consensus_queue.cc:260] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } }
I20240501 07:58:57.510558 714 raft_consensus.cc:3037] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:57.511224 2424 raft_consensus.cc:2454] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36272be7c71b413a94d36b4257d34425 in term 2.
I20240501 07:58:57.511257 2293 raft_consensus.cc:2454] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36272be7c71b413a94d36b4257d34425 in term 2.
I20240501 07:58:57.511525 714 raft_consensus.cc:2454] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36272be7c71b413a94d36b4257d34425 in term 2.
I20240501 07:58:57.511610 2097 leader_election.cc:302] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 36272be7c71b413a94d36b4257d34425, 6645111fc9e446e8877750a3c4779750, c1b68243a7144af5835b051aa9916d13; no voters:
I20240501 07:58:57.511803 4175 raft_consensus.cc:2781] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:57.512013 4175 raft_consensus.cc:686] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [term 2 LEADER]: Becoming Leader. State: Replica: 36272be7c71b413a94d36b4257d34425, State: Running, Role: LEADER
I20240501 07:58:57.512194 4175 consensus_queue.cc:237] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } }
I20240501 07:58:57.514395 433 catalog_manager.cc:5484] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 reported cstate change: term changed from 1 to 2, leader changed from e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130) to 36272be7c71b413a94d36b4257d34425 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "36272be7c71b413a94d36b4257d34425" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 07:58:57.596144 910 consensus_peers.cc:485] T 46ad498fe543450f8911b83dd71d07f3 P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:58:57.743646 648 consensus_peers.cc:485] T 5ab169c1536c4b849918c7cd593048c1 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 07:58:57.896235 714 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 2 FOLLOWER]: Refusing update from remote peer 36272be7c71b413a94d36b4257d34425: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 07:58:57.896733 4175 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 07:58:58.023792 779 consensus_peers.cc:485] T d6e4d59526a048fb8bb06d8e4a9de604 P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 07:58:58.044307 2424 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Refusing update from remote peer 36272be7c71b413a94d36b4257d34425: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 07:58:58.044802 4175 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 07:58:58.046948 648 consensus_peers.cc:485] T 789ccd2b398d450884458b02e51039e7 P e2c804beb8ef40ecb16c952f8875ca02 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 07:58:58.049252 2293 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13 [term 2 FOLLOWER]: Refusing update from remote peer 36272be7c71b413a94d36b4257d34425: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 07:58:58.049763 4264 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 07:58:58.050341 779 consensus_peers.cc:485] T 4f220c7dad9d4cf390608b13c5de0fbf P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:58:58.350636 910 consensus_peers.cc:485] T 2c969b6e50fd4fd4919c181ae51a47ae P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 07:58:58.522260 2162 consensus_queue.cc:237] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } }
I20240501 07:58:58.523444 2293 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13 [term 2 FOLLOWER]: Refusing update from remote peer 36272be7c71b413a94d36b4257d34425: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 07:58:58.523537 2424 raft_consensus.cc:1261] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Refusing update from remote peer 36272be7c71b413a94d36b4257d34425: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 07:58:58.523859 4175 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 07:58:58.524055 4262 consensus_queue.cc:1035] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 07:58:58.525888 4264 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P 36272be7c71b413a94d36b4257d34425 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:58.526358 2424 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:58.526355 2293 raft_consensus.cc:2932] T b8e9192810694fa1bf54ca88fb619d11 P c1b68243a7144af5835b051aa9916d13 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:58.527704 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b8e9192810694fa1bf54ca88fb619d11 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:58:58.529170 433 catalog_manager.cc:5484] T b8e9192810694fa1bf54ca88fb619d11 P 6645111fc9e446e8877750a3c4779750 reported cstate change: config changed from index 11 to 13, VOTER e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "36272be7c71b413a94d36b4257d34425" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: false } } peers { permanent_uuid: "c1b68243a7144af5835b051aa9916d13" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 44501 } attrs { promote: false } } peers { permanent_uuid: "36272be7c71b413a94d36b4257d34425" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 39945 } attrs { promote: false } } }
I20240501 07:58:58.532461 692 tablet_service.cc:1514] Processing DeleteTablet for tablet b8e9192810694fa1bf54ca88fb619d11 with delete_type TABLET_DATA_TOMBSTONED (TS e2c804beb8ef40ecb16c952f8875ca02 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:43240
I20240501 07:58:58.532845 4283 tablet_replica.cc:330] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02: stopping tablet replica
I20240501 07:58:58.533025 4283 raft_consensus.cc:2229] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:58.533139 4283 raft_consensus.cc:2258] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:58.533722 4283 ts_tablet_manager.cc:1875] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:58.535668 4283 ts_tablet_manager.cc:1888] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240501 07:58:58.535737 4283 log.cc:1196] T b8e9192810694fa1bf54ca88fb619d11 P e2c804beb8ef40ecb16c952f8875ca02: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b8e9192810694fa1bf54ca88fb619d11
I20240501 07:58:58.536293 426 catalog_manager.cc:4830] TS e2c804beb8ef40ecb16c952f8875ca02 (127.0.99.130:43577): tablet b8e9192810694fa1bf54ca88fb619d11 (table rebalance_test_table_1 [id=e08d5aefd3334c55a539f0f47df899e4]) successfully deleted
I20240501 07:58:58.737004 2404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:58.737860 2142 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:58.738611 825 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:58.738684 2011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:58.746081 692 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:58.750206 955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 07:58:58.760123 910 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240501 07:58:58.769910 2273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 07:58:58.822038 779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240501 07:58:58.865084 2142 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:58.866060 825 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:58.866068 692 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:58.866477 2011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:58.867223 2273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:58.867837 955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:58.868237 2404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:58.925670 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:35849
--local_ip_for_outbound_sockets=127.0.99.129
--tserver_master_addrs=127.0.99.190:35609
--webserver_port=42113
--webserver_interface=127.0.99.129
--builtin_ntp_servers=127.0.99.148:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 07:58:59.042110 4360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:59.042357 4360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:59.042407 4360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:59.044086 910 consensus_peers.cc:485] T a25b951ae4804bc7bd1a8f1ef58e2bde P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:58:59.045782 4360 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:59.045866 4360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:59.045971 4360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240501 07:58:59.050128 4360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:35849
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=42113
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35609
--never_fsync=true
--heap_profile_path=/tmp/kudu.4360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:59.051142 4360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:59.052309 4360 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:59.061542 4368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:59.062031 4366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:59.062602 4365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:59.063323 4360 server_base.cc:1034] running on GCE node
I20240501 07:58:59.063992 4360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:59.064996 4360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:59.066205 4360 hybrid_clock.cc:648] HybridClock initialized: now 1714550339066173 us; error 72 us; skew 500 ppm
I20240501 07:58:59.068529 4360 webserver.cc:460] Webserver started at http://127.0.99.129:42113/ using document root <none> and password file <none>
I20240501 07:58:59.069142 4360 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:59.069231 4360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:59.073930 4360 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:59.077049 4374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:59.078265 4360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 07:58:59.078400 4360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-2srn"
I20240501 07:58:59.078927 4360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:59.098482 4360 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:59.099421 4360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:59.099634 4360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:59.100248 4360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:59.101605 4381 ts_tablet_manager.cc:516] Loading tablet metadata (0/12 complete)
W20240501 07:58:59.105377 648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:35849: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240501 07:58:59.114777 4360 ts_tablet_manager.cc:559] Loaded tablet metadata (12 total tablets, 12 live tablets)
I20240501 07:58:59.114883 4360 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.000s
I20240501 07:58:59.114959 4360 ts_tablet_manager.cc:574] Registering tablets (0/12 complete)
I20240501 07:58:59.117468 4381 tablet_bootstrap.cc:492] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:59.121907 4360 ts_tablet_manager.cc:590] Registered 12 tablets
I20240501 07:58:59.121974 4360 ts_tablet_manager.cc:569] Time spent register tablets: real 0.007s user 0.004s sys 0.000s
I20240501 07:58:59.132082 4360 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:35849
I20240501 07:58:59.132153 4488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:35849 every 8 connection(s)
I20240501 07:58:59.133296 4360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 07:58:59.133572 4381 log.cc:824] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:59.142565 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 4360
I20240501 07:58:59.142650 4489 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35609
I20240501 07:58:59.143035 4489 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:59.143707 4489 heartbeater.cc:511] Master 127.0.99.190:35609 requested a full tablet report, sending...
I20240501 07:58:59.146047 436 ts_manager.cc:194] Re-registered known tserver with Master: 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849)
I20240501 07:58:59.148767 436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:47269
I20240501 07:58:59.154590 4381 tablet_bootstrap.cc:492] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:59.155077 4381 tablet_bootstrap.cc:492] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap complete.
I20240501 07:58:59.156559 4381 ts_tablet_manager.cc:1367] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.039s user 0.032s sys 0.004s
I20240501 07:58:59.159489 4381 raft_consensus.cc:348] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:59.159884 4381 raft_consensus.cc:729] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.160399 4381 consensus_queue.cc:260] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:59.161221 4489 heartbeater.cc:503] Master 127.0.99.190:35609 was elected leader, sending a full tablet report...
I20240501 07:58:59.161357 4381 ts_tablet_manager.cc:1398] T 2c969b6e50fd4fd4919c181ae51a47ae P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 07:58:59.161614 4381 tablet_bootstrap.cc:492] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:59.188273 4381 tablet_bootstrap.cc:492] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:59.188663 4381 tablet_bootstrap.cc:492] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap complete.
I20240501 07:58:59.189640 4381 ts_tablet_manager.cc:1367] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.004s
I20240501 07:58:59.190191 4381 raft_consensus.cc:348] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:59.190310 4381 raft_consensus.cc:729] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.190469 4381 consensus_queue.cc:260] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:59.190732 4381 ts_tablet_manager.cc:1398] T 88887f8c213f46b2b4d4555d6a13c346 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 07:58:59.190909 4381 tablet_bootstrap.cc:492] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
W20240501 07:58:59.194159 779 consensus_peers.cc:485] T c6f89045c20f4ccbb525febd8f5ce56e P 10aab83192f447829c9b8254e36d94b4 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 41: this message will repeat every 5th retry.
I20240501 07:58:59.217754 4381 tablet_bootstrap.cc:492] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:59.218153 4381 tablet_bootstrap.cc:492] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap complete.
I20240501 07:58:59.218978 4381 ts_tablet_manager.cc:1367] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.015s
I20240501 07:58:59.219419 4381 raft_consensus.cc:348] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } }
I20240501 07:58:59.219539 4381 raft_consensus.cc:729] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.219677 4381 consensus_queue.cc:260] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } attrs { replace: true } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "6645111fc9e446e8877750a3c4779750" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 44643 } attrs { promote: true } }
I20240501 07:58:59.219944 4381 ts_tablet_manager.cc:1398] T 9d5700ab4d4f465bbe08ad25d0b9084f P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:59.220094 4381 tablet_bootstrap.cc:492] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:59.245834 4381 tablet_bootstrap.cc:492] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:59.246228 4381 tablet_bootstrap.cc:492] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap complete.
I20240501 07:58:59.247164 4381 ts_tablet_manager.cc:1367] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.027s user 0.019s sys 0.007s
I20240501 07:58:59.247663 4381 raft_consensus.cc:348] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:59.247773 4381 raft_consensus.cc:729] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.247907 4381 consensus_queue.cc:260] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:59.248167 4381 ts_tablet_manager.cc:1398] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:59.248313 4381 tablet_bootstrap.cc:492] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:59.274804 4381 tablet_bootstrap.cc:492] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:59.275276 4381 tablet_bootstrap.cc:492] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap complete.
I20240501 07:58:59.276002 4381 ts_tablet_manager.cc:1367] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.028s user 0.022s sys 0.004s
I20240501 07:58:59.276427 4381 raft_consensus.cc:348] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:59.276532 4381 raft_consensus.cc:729] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.276674 4381 consensus_queue.cc:260] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:59.277117 4381 ts_tablet_manager.cc:1398] T c6f89045c20f4ccbb525febd8f5ce56e P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:59.277624 4381 tablet_bootstrap.cc:492] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
W20240501 07:58:59.300520 910 consensus_peers.cc:485] T 5d48c8225721471f9eea917078d5ef2d P e2edc87d834f4e0bb662d501af69f2b5 -> Peer 2c148dc68b5a4a38ba10be4e1e4b8cd2 (127.0.99.129:35849): Couldn't send request to peer 2c148dc68b5a4a38ba10be4e1e4b8cd2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 41: this message will repeat every 5th retry.
I20240501 07:58:59.305599 4381 tablet_bootstrap.cc:492] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:59.305984 4381 tablet_bootstrap.cc:492] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap complete.
I20240501 07:58:59.307076 4381 ts_tablet_manager.cc:1367] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.011s
I20240501 07:58:59.307554 4381 raft_consensus.cc:348] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:59.307680 4381 raft_consensus.cc:729] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.307806 4381 consensus_queue.cc:260] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:59.308081 4381 ts_tablet_manager.cc:1398] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:59.308228 4381 tablet_bootstrap.cc:492] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:59.338351 4381 tablet_bootstrap.cc:492] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:59.338712 4381 tablet_bootstrap.cc:492] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap complete.
I20240501 07:58:59.339656 4381 ts_tablet_manager.cc:1367] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.031s user 0.026s sys 0.004s
I20240501 07:58:59.340127 4381 raft_consensus.cc:348] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:59.340269 4381 raft_consensus.cc:729] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.340411 4381 consensus_queue.cc:260] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } }
I20240501 07:58:59.340823 4381 ts_tablet_manager.cc:1398] T 5ab169c1536c4b849918c7cd593048c1 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:59.340993 4381 tablet_bootstrap.cc:492] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:59.371080 4381 tablet_bootstrap.cc:492] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:59.371470 4381 tablet_bootstrap.cc:492] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap complete.
I20240501 07:58:59.372443 4381 ts_tablet_manager.cc:1367] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.008s
I20240501 07:58:59.372928 4381 raft_consensus.cc:348] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:59.373064 4381 raft_consensus.cc:729] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.373208 4381 consensus_queue.cc:260] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } }
I20240501 07:58:59.373485 4381 ts_tablet_manager.cc:1398] T e7eefec740384c688dc6e149b5ea2769 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:59.373647 4381 tablet_bootstrap.cc:492] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:59.394228 2142 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:59.394601 825 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:59.394594 4423 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:59.404963 4381 tablet_bootstrap.cc:492] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:59.405368 4381 tablet_bootstrap.cc:492] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap complete.
I20240501 07:58:59.406314 4381 ts_tablet_manager.cc:1367] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.012s
I20240501 07:58:59.407300 4381 raft_consensus.cc:348] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:59.407428 4381 raft_consensus.cc:729] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.408488 692 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:59.408445 4381 consensus_queue.cc:260] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:59.411477 4381 ts_tablet_manager.cc:1398] T a25b951ae4804bc7bd1a8f1ef58e2bde P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:59.413656 4381 tablet_bootstrap.cc:492] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:59.425302 2011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:59.431313 2273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:59.434324 2404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:59.449513 955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:59.452873 4381 tablet_bootstrap.cc:492] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:59.453272 4381 tablet_bootstrap.cc:492] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap complete.
I20240501 07:58:59.455009 4381 ts_tablet_manager.cc:1367] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.041s user 0.024s sys 0.010s
I20240501 07:58:59.455966 4381 raft_consensus.cc:348] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:59.456221 4381 raft_consensus.cc:729] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.456527 4381 consensus_queue.cc:260] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } } peers { permanent_uuid: "e2c804beb8ef40ecb16c952f8875ca02" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43577 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } }
I20240501 07:58:59.457151 4381 ts_tablet_manager.cc:1398] T 789ccd2b398d450884458b02e51039e7 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 07:58:59.457571 4381 tablet_bootstrap.cc:492] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:59.485445 4381 tablet_bootstrap.cc:492] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:59.485793 4381 tablet_bootstrap.cc:492] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap complete.
I20240501 07:58:59.486770 4381 ts_tablet_manager.cc:1367] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.029s user 0.021s sys 0.007s
I20240501 07:58:59.487246 4381 raft_consensus.cc:348] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:59.487372 4381 raft_consensus.cc:729] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.487509 4381 consensus_queue.cc:260] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:59.487788 4381 ts_tablet_manager.cc:1398] T 46ad498fe543450f8911b83dd71d07f3 P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:59.487944 4381 tablet_bootstrap.cc:492] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap starting.
I20240501 07:58:59.520289 4381 tablet_bootstrap.cc:492] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:59.520628 4381 tablet_bootstrap.cc:492] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Bootstrap complete.
I20240501 07:58:59.521522 4381 ts_tablet_manager.cc:1367] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.007s
I20240501 07:58:59.521941 4381 raft_consensus.cc:348] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:59.522076 4381 raft_consensus.cc:729] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c148dc68b5a4a38ba10be4e1e4b8cd2, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.522205 4381 consensus_queue.cc:260] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10aab83192f447829c9b8254e36d94b4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40475 } } peers { permanent_uuid: "e2edc87d834f4e0bb662d501af69f2b5" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36973 } } peers { permanent_uuid: "2c148dc68b5a4a38ba10be4e1e4b8cd2" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35849 } }
I20240501 07:58:59.522519 4381 ts_tablet_manager.cc:1398] T 5d48c8225721471f9eea917078d5ef2d P 2c148dc68b5a4a38ba10be4e1e4b8cd2: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20240501 07:58:59.528237 4443 raft_consensus.cc:3037] T 4f220c7dad9d4cf390608b13c5de0fbf P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:59.531296 4442 raft_consensus.cc:3037] T d6e4d59526a048fb8bb06d8e4a9de604 P 2c148dc68b5a4a38ba10be4e1e4b8cd2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:59.563124 825 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:58:59.564330 4423 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:59.564781 2011 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:59.568337 2142 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:59.585567 2273 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:59.591146 955 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:58:59.610074 2404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:59.618572 692 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:59.650101 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 4360
I20240501 07:58:59.656196 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 632
I20240501 07:58:59.662307 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 763
I20240501 07:58:59.668344 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 894
I20240501 07:58:59.674358 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 1949
I20240501 07:58:59.680145 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 2080
I20240501 07:58:59.685763 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 2211
I20240501 07:58:59.691478 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 2341
I20240501 07:58:59.696985 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 410
2024-05-01T07:58:59Z chronyd exiting
[ OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (33067 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (33067 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240501 07:58:59.728042 398 test_util.cc:268] Using random seed: -604835166
I20240501 07:58:59.728884 398 ts_itest-base.cc:112] Starting cluster with:
I20240501 07:58:59.728936 398 ts_itest-base.cc:113] --------------
I20240501 07:58:59.728947 398 ts_itest-base.cc:114] 5 tablet servers
I20240501 07:58:59.728958 398 ts_itest-base.cc:115] 3 replicas per TS
I20240501 07:58:59.728968 398 ts_itest-base.cc:116] --------------
2024-05-01T07:58:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T07:58:59Z Disabled control of system clock
I20240501 07:58:59.737656 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:37191
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:37191
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 07:58:59.851789 4604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:59.852072 4604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:59.852133 4604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:59.855381 4604 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:58:59.855480 4604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 07:58:59.855509 4604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:59.855535 4604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 07:58:59.855567 4604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 07:58:59.859537 4604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:37191
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:37191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.4604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:59.860534 4604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:59.861660 4604 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:59.869463 4610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:59.869913 4609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:59.870545 4612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:59.870997 4604 server_base.cc:1034] running on GCE node
I20240501 07:58:59.871588 4604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:59.872642 4604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:59.873873 4604 hybrid_clock.cc:648] HybridClock initialized: now 1714550339873860 us; error 31 us; skew 500 ppm
I20240501 07:58:59.875990 4604 webserver.cc:460] Webserver started at http://127.0.99.190:34249/ using document root <none> and password file <none>
I20240501 07:58:59.876561 4604 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:59.876621 4604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:59.876796 4604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:59.878541 4604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8622e81d89344034ac44f4b7aeb7c49b"
format_stamp: "Formatted at 2024-05-01 07:58:59 on dist-test-slave-2srn"
I20240501 07:58:59.879029 4604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8622e81d89344034ac44f4b7aeb7c49b"
format_stamp: "Formatted at 2024-05-01 07:58:59 on dist-test-slave-2srn"
I20240501 07:58:59.882632 4604 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240501 07:58:59.885037 4618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:59.886174 4604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:59.886323 4604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8622e81d89344034ac44f4b7aeb7c49b"
format_stamp: "Formatted at 2024-05-01 07:58:59 on dist-test-slave-2srn"
I20240501 07:58:59.886479 4604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:59.903554 4604 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:59.904260 4604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:59.904445 4604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:59.911656 4604 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:37191
I20240501 07:58:59.911666 4670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:37191 every 8 connection(s)
I20240501 07:58:59.912720 4604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 07:58:59.913586 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 4604
I20240501 07:58:59.913710 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 07:58:59.915977 4671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.920861 4671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b: Bootstrap starting.
I20240501 07:58:59.923238 4671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.924015 4671 log.cc:824] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:59.925799 4671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b: No bootstrap required, opened a new log
I20240501 07:58:59.928582 4671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8622e81d89344034ac44f4b7aeb7c49b" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37191 } }
I20240501 07:58:59.928791 4671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.928854 4671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8622e81d89344034ac44f4b7aeb7c49b, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.929278 4671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8622e81d89344034ac44f4b7aeb7c49b" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37191 } }
I20240501 07:58:59.929419 4671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:59.929478 4671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:59.929584 4671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.930410 4671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8622e81d89344034ac44f4b7aeb7c49b" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37191 } }
I20240501 07:58:59.930809 4671 leader_election.cc:302] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8622e81d89344034ac44f4b7aeb7c49b; no voters:
I20240501 07:58:59.931099 4671 leader_election.cc:288] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:59.931221 4676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:59.931429 4676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b [term 1 LEADER]: Becoming Leader. State: Replica: 8622e81d89344034ac44f4b7aeb7c49b, State: Running, Role: LEADER
I20240501 07:58:59.932063 4676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8622e81d89344034ac44f4b7aeb7c49b" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37191 } }
I20240501 07:58:59.932210 4671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:58:59.933542 4678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8622e81d89344034ac44f4b7aeb7c49b. Latest consensus state: current_term: 1 leader_uuid: "8622e81d89344034ac44f4b7aeb7c49b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8622e81d89344034ac44f4b7aeb7c49b" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37191 } } }
I20240501 07:58:59.933679 4678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:59.933962 4677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8622e81d89344034ac44f4b7aeb7c49b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8622e81d89344034ac44f4b7aeb7c49b" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37191 } } }
I20240501 07:58:59.934031 4686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:58:59.934064 4677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:59.936918 4686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:58:59.942179 4686 catalog_manager.cc:1347] Generated new cluster ID: a2496dddd6394e1e9919c450ebe7774a
I20240501 07:58:59.942277 4686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:58:59.949371 4686 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:58:59.950383 4686 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:58:59.962356 4686 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8622e81d89344034ac44f4b7aeb7c49b: Generated new TSK 0
I20240501 07:58:59.963194 4686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:58:59.967983 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37191
--builtin_ntp_servers=127.0.99.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 07:59:00.078446 4695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:00.078712 4695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:00.078775 4695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:00.081955 4695 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:00.082052 4695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:00.082147 4695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240501 07:59:00.086045 4695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37191
--never_fsync=true
--heap_profile_path=/tmp/kudu.4695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:00.087039 4695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:00.088186 4695 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:00.097146 4700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:00.097503 4701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:00.098163 4703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:00.098704 4695 server_base.cc:1034] running on GCE node
I20240501 07:59:00.099159 4695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:00.099756 4695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:00.100914 4695 hybrid_clock.cc:648] HybridClock initialized: now 1714550340100898 us; error 31 us; skew 500 ppm
I20240501 07:59:00.103039 4695 webserver.cc:460] Webserver started at http://127.0.99.129:40285/ using document root <none> and password file <none>
I20240501 07:59:00.103605 4695 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:00.103663 4695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:00.103839 4695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:00.105566 4695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f886adf6ee224ebfb3c783d93ef386e1"
format_stamp: "Formatted at 2024-05-01 07:59:00 on dist-test-slave-2srn"
I20240501 07:59:00.106066 4695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f886adf6ee224ebfb3c783d93ef386e1"
format_stamp: "Formatted at 2024-05-01 07:59:00 on dist-test-slave-2srn"
I20240501 07:59:00.109566 4695 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240501 07:59:00.112118 4709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:00.113183 4695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:00.113344 4695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f886adf6ee224ebfb3c783d93ef386e1"
format_stamp: "Formatted at 2024-05-01 07:59:00 on dist-test-slave-2srn"
I20240501 07:59:00.113451 4695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:00.126986 4695 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:00.127665 4695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:00.127839 4695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:00.128386 4695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:00.129335 4695 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:00.129382 4695 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:00.129436 4695 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:00.129462 4695 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:00.138075 4695 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:40291
I20240501 07:59:00.138100 4822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:40291 every 8 connection(s)
I20240501 07:59:00.139065 4695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 07:59:00.144275 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 4695
I20240501 07:59:00.144395 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 07:59:00.146634 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37191
--builtin_ntp_servers=127.0.99.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:00.148836 4823 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37191
I20240501 07:59:00.149088 4823 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:00.149621 4823 heartbeater.cc:511] Master 127.0.99.190:37191 requested a full tablet report, sending...
I20240501 07:59:00.151449 4635 ts_manager.cc:194] Registered new tserver with Master: f886adf6ee224ebfb3c783d93ef386e1 (127.0.99.129:40291)
I20240501 07:59:00.153174 4635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:44977
W20240501 07:59:00.257769 4826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:00.258044 4826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:00.258091 4826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:00.261343 4826 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:00.261441 4826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:00.261515 4826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240501 07:59:00.265666 4826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37191
--never_fsync=true
--heap_profile_path=/tmp/kudu.4826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:00.266707 4826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:00.267997 4826 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:00.277702 4832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:00.278193 4834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:00.278818 4831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:00.279409 4826 server_base.cc:1034] running on GCE node
I20240501 07:59:00.279891 4826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:00.280529 4826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:00.281718 4826 hybrid_clock.cc:648] HybridClock initialized: now 1714550340281694 us; error 41 us; skew 500 ppm
I20240501 07:59:00.283988 4826 webserver.cc:460] Webserver started at http://127.0.99.130:41095/ using document root <none> and password file <none>
I20240501 07:59:00.284576 4826 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:00.284641 4826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:00.284853 4826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:00.286715 4826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "12da6fa02ac940bea357b120f3d1029a"
format_stamp: "Formatted at 2024-05-01 07:59:00 on dist-test-slave-2srn"
I20240501 07:59:00.287228 4826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "12da6fa02ac940bea357b120f3d1029a"
format_stamp: "Formatted at 2024-05-01 07:59:00 on dist-test-slave-2srn"
I20240501 07:59:00.291297 4826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 07:59:00.294116 4840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:00.295384 4826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 07:59:00.295607 4826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "12da6fa02ac940bea357b120f3d1029a"
format_stamp: "Formatted at 2024-05-01 07:59:00 on dist-test-slave-2srn"
I20240501 07:59:00.295743 4826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:00.312234 4826 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:00.312951 4826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:00.313136 4826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:00.313719 4826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:00.314743 4826 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:00.314800 4826 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:00.314862 4826 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:00.314891 4826 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:00.324540 4953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:38837 every 8 connection(s)
I20240501 07:59:00.324518 4826 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:38837
I20240501 07:59:00.325614 4826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 07:59:00.333530 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 4826
I20240501 07:59:00.333643 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 07:59:00.336385 4954 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37191
I20240501 07:59:00.336649 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37191
--builtin_ntp_servers=127.0.99.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:00.336679 4954 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:00.337265 4954 heartbeater.cc:511] Master 127.0.99.190:37191 requested a full tablet report, sending...
I20240501 07:59:00.338356 4635 ts_manager.cc:194] Registered new tserver with Master: 12da6fa02ac940bea357b120f3d1029a (127.0.99.130:38837)
I20240501 07:59:00.338963 4635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:34875
W20240501 07:59:00.455852 4957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:00.456117 4957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:00.456163 4957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:00.459385 4957 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:00.459475 4957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:00.459547 4957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240501 07:59:00.463482 4957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37191
--never_fsync=true
--heap_profile_path=/tmp/kudu.4957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:00.464579 4957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:00.465766 4957 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:00.475230 4962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:00.475793 4965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:00.476347 4963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:00.476929 4957 server_base.cc:1034] running on GCE node
I20240501 07:59:00.477342 4957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:00.477944 4957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:00.479118 4957 hybrid_clock.cc:648] HybridClock initialized: now 1714550340479091 us; error 44 us; skew 500 ppm
I20240501 07:59:00.481221 4957 webserver.cc:460] Webserver started at http://127.0.99.131:43091/ using document root <none> and password file <none>
I20240501 07:59:00.481783 4957 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:00.481840 4957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:00.482026 4957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:00.483768 4957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3845815f54b54219aaa5f8d8064ba050"
format_stamp: "Formatted at 2024-05-01 07:59:00 on dist-test-slave-2srn"
I20240501 07:59:00.484246 4957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3845815f54b54219aaa5f8d8064ba050"
format_stamp: "Formatted at 2024-05-01 07:59:00 on dist-test-slave-2srn"
I20240501 07:59:00.487730 4957 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:59:00.490300 4971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:00.491523 4957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:00.491662 4957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3845815f54b54219aaa5f8d8064ba050"
format_stamp: "Formatted at 2024-05-01 07:59:00 on dist-test-slave-2srn"
I20240501 07:59:00.491765 4957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:00.508867 4957 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:00.509529 4957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:00.509706 4957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:00.510267 4957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:00.511229 4957 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:00.511278 4957 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:00.511332 4957 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:00.511360 4957 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:00.519637 4957 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:42867
I20240501 07:59:00.519656 5084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:42867 every 8 connection(s)
I20240501 07:59:00.520581 4957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 07:59:00.522802 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 4957
I20240501 07:59:00.522886 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 07:59:00.524889 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37191
--builtin_ntp_servers=127.0.99.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:00.530463 5085 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37191
I20240501 07:59:00.530732 5085 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:00.531241 5085 heartbeater.cc:511] Master 127.0.99.190:37191 requested a full tablet report, sending...
I20240501 07:59:00.532564 4635 ts_manager.cc:194] Registered new tserver with Master: 3845815f54b54219aaa5f8d8064ba050 (127.0.99.131:42867)
I20240501 07:59:00.533488 4635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:47171
W20240501 07:59:00.636404 5088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:00.636669 5088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:00.636714 5088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:00.639890 5088 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:00.639969 5088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:00.640080 5088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:59:00.643954 5088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37191
--never_fsync=true
--heap_profile_path=/tmp/kudu.5088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:00.644963 5088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:00.646107 5088 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:00.654844 5096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:00.655256 5093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:00.655792 5094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:00.656286 5088 server_base.cc:1034] running on GCE node
I20240501 07:59:00.656703 5088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:00.657317 5088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:00.658474 5088 hybrid_clock.cc:648] HybridClock initialized: now 1714550340658456 us; error 41 us; skew 500 ppm
I20240501 07:59:00.660506 5088 webserver.cc:460] Webserver started at http://127.0.99.132:32849/ using document root <none> and password file <none>
I20240501 07:59:00.661055 5088 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:00.661110 5088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:00.661285 5088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:00.663035 5088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4d38ff60089547989a10843c41f76a03"
format_stamp: "Formatted at 2024-05-01 07:59:00 on dist-test-slave-2srn"
I20240501 07:59:00.663514 5088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4d38ff60089547989a10843c41f76a03"
format_stamp: "Formatted at 2024-05-01 07:59:00 on dist-test-slave-2srn"
I20240501 07:59:00.667083 5088 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:59:00.669445 5102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:00.670672 5088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:00.670790 5088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4d38ff60089547989a10843c41f76a03"
format_stamp: "Formatted at 2024-05-01 07:59:00 on dist-test-slave-2srn"
I20240501 07:59:00.670897 5088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:00.680989 5088 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:00.681670 5088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:00.681826 5088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:00.682375 5088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:00.683339 5088 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:00.683388 5088 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:00.683432 5088 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:00.683459 5088 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:00.692330 5088 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:33539
I20240501 07:59:00.692342 5215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:33539 every 8 connection(s)
I20240501 07:59:00.693359 5088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:59:00.700791 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 5088
I20240501 07:59:00.700909 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 07:59:00.703233 5216 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37191
I20240501 07:59:00.703343 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37191
--builtin_ntp_servers=127.0.99.148:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:00.703511 5216 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:00.704007 5216 heartbeater.cc:511] Master 127.0.99.190:37191 requested a full tablet report, sending...
I20240501 07:59:00.705013 4635 ts_manager.cc:194] Registered new tserver with Master: 4d38ff60089547989a10843c41f76a03 (127.0.99.132:33539)
I20240501 07:59:00.705595 4635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:41967
W20240501 07:59:00.817274 5219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:00.817545 5219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:00.817605 5219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:00.820873 5219 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:00.820959 5219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:00.821036 5219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:59:00.825150 5219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37191
--never_fsync=true
--heap_profile_path=/tmp/kudu.5219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:00.826175 5219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:00.827375 5219 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:00.836495 5224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:00.837028 5225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:00.837481 5227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:00.838110 5219 server_base.cc:1034] running on GCE node
I20240501 07:59:00.838619 5219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:00.839277 5219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:00.840456 5219 hybrid_clock.cc:648] HybridClock initialized: now 1714550340840439 us; error 32 us; skew 500 ppm
I20240501 07:59:00.842649 5219 webserver.cc:460] Webserver started at http://127.0.99.133:39123/ using document root <none> and password file <none>
I20240501 07:59:00.843226 5219 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:00.843287 5219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:00.843468 5219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:00.845364 5219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "cd24435aa8644984805bdbf02803ddd8"
format_stamp: "Formatted at 2024-05-01 07:59:00 on dist-test-slave-2srn"
I20240501 07:59:00.845945 5219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "cd24435aa8644984805bdbf02803ddd8"
format_stamp: "Formatted at 2024-05-01 07:59:00 on dist-test-slave-2srn"
I20240501 07:59:00.849853 5219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20240501 07:59:00.852324 5233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:00.853514 5219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:00.853642 5219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "cd24435aa8644984805bdbf02803ddd8"
format_stamp: "Formatted at 2024-05-01 07:59:00 on dist-test-slave-2srn"
I20240501 07:59:00.853744 5219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:00.874574 5219 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:00.875217 5219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:00.875372 5219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:00.875905 5219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:00.876786 5219 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:00.876833 5219 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:00.876888 5219 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:00.876912 5219 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:00.885206 5219 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:37633
I20240501 07:59:00.885226 5346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:37633 every 8 connection(s)
I20240501 07:59:00.886158 5219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:59:00.890213 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 5219
I20240501 07:59:00.890326 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 07:59:00.895490 5347 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37191
I20240501 07:59:00.895745 5347 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:00.896221 5347 heartbeater.cc:511] Master 127.0.99.190:37191 requested a full tablet report, sending...
I20240501 07:59:00.897154 4635 ts_manager.cc:194] Registered new tserver with Master: cd24435aa8644984805bdbf02803ddd8 (127.0.99.133:37633)
I20240501 07:59:00.897716 4635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:53881
I20240501 07:59:00.904589 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 07:59:00.914930 4635 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52574:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 07:59:00.933559 5019 tablet_service.cc:1467] Processing CreateTablet for tablet a6c16f2cb2074e02acb65dcde42c0a09 (DEFAULT_TABLE table=TestTable [id=9fd48fd40b1149bb96dbc3540b99a08a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.934650 5019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c16f2cb2074e02acb65dcde42c0a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.934692 4888 tablet_service.cc:1467] Processing CreateTablet for tablet a6c16f2cb2074e02acb65dcde42c0a09 (DEFAULT_TABLE table=TestTable [id=9fd48fd40b1149bb96dbc3540b99a08a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.935286 5281 tablet_service.cc:1467] Processing CreateTablet for tablet a6c16f2cb2074e02acb65dcde42c0a09 (DEFAULT_TABLE table=TestTable [id=9fd48fd40b1149bb96dbc3540b99a08a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.935683 4888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c16f2cb2074e02acb65dcde42c0a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.936168 5281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c16f2cb2074e02acb65dcde42c0a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.940039 5364 tablet_bootstrap.cc:492] T a6c16f2cb2074e02acb65dcde42c0a09 P 3845815f54b54219aaa5f8d8064ba050: Bootstrap starting.
I20240501 07:59:00.941915 5365 tablet_bootstrap.cc:492] T a6c16f2cb2074e02acb65dcde42c0a09 P 12da6fa02ac940bea357b120f3d1029a: Bootstrap starting.
I20240501 07:59:00.942546 5364 tablet_bootstrap.cc:654] T a6c16f2cb2074e02acb65dcde42c0a09 P 3845815f54b54219aaa5f8d8064ba050: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.943374 5366 tablet_bootstrap.cc:492] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8: Bootstrap starting.
I20240501 07:59:00.943374 5364 log.cc:824] T a6c16f2cb2074e02acb65dcde42c0a09 P 3845815f54b54219aaa5f8d8064ba050: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:00.945096 5364 tablet_bootstrap.cc:492] T a6c16f2cb2074e02acb65dcde42c0a09 P 3845815f54b54219aaa5f8d8064ba050: No bootstrap required, opened a new log
I20240501 07:59:00.945277 5364 ts_tablet_manager.cc:1367] T a6c16f2cb2074e02acb65dcde42c0a09 P 3845815f54b54219aaa5f8d8064ba050: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 07:59:00.945464 5365 tablet_bootstrap.cc:654] T a6c16f2cb2074e02acb65dcde42c0a09 P 12da6fa02ac940bea357b120f3d1029a: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.945464 5366 tablet_bootstrap.cc:654] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.946359 5366 log.cc:824] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:00.946359 5365 log.cc:824] T a6c16f2cb2074e02acb65dcde42c0a09 P 12da6fa02ac940bea357b120f3d1029a: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:00.948173 5365 tablet_bootstrap.cc:492] T a6c16f2cb2074e02acb65dcde42c0a09 P 12da6fa02ac940bea357b120f3d1029a: No bootstrap required, opened a new log
I20240501 07:59:00.948174 5366 tablet_bootstrap.cc:492] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8: No bootstrap required, opened a new log
I20240501 07:59:00.948371 5366 ts_tablet_manager.cc:1367] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:59:00.948371 5365 ts_tablet_manager.cc:1367] T a6c16f2cb2074e02acb65dcde42c0a09 P 12da6fa02ac940bea357b120f3d1029a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 07:59:00.948423 5364 raft_consensus.cc:348] T a6c16f2cb2074e02acb65dcde42c0a09 P 3845815f54b54219aaa5f8d8064ba050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12da6fa02ac940bea357b120f3d1029a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38837 } } peers { permanent_uuid: "cd24435aa8644984805bdbf02803ddd8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37633 } } peers { permanent_uuid: "3845815f54b54219aaa5f8d8064ba050" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42867 } }
I20240501 07:59:00.948652 5364 raft_consensus.cc:374] T a6c16f2cb2074e02acb65dcde42c0a09 P 3845815f54b54219aaa5f8d8064ba050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.948707 5364 raft_consensus.cc:729] T a6c16f2cb2074e02acb65dcde42c0a09 P 3845815f54b54219aaa5f8d8064ba050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3845815f54b54219aaa5f8d8064ba050, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.949106 5364 consensus_queue.cc:260] T a6c16f2cb2074e02acb65dcde42c0a09 P 3845815f54b54219aaa5f8d8064ba050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12da6fa02ac940bea357b120f3d1029a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38837 } } peers { permanent_uuid: "cd24435aa8644984805bdbf02803ddd8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37633 } } peers { permanent_uuid: "3845815f54b54219aaa5f8d8064ba050" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42867 } }
I20240501 07:59:00.949911 5364 ts_tablet_manager.cc:1398] T a6c16f2cb2074e02acb65dcde42c0a09 P 3845815f54b54219aaa5f8d8064ba050: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20240501 07:59:00.950255 5085 heartbeater.cc:503] Master 127.0.99.190:37191 was elected leader, sending a full tablet report...
I20240501 07:59:00.952309 5366 raft_consensus.cc:348] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12da6fa02ac940bea357b120f3d1029a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38837 } } peers { permanent_uuid: "cd24435aa8644984805bdbf02803ddd8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37633 } } peers { permanent_uuid: "3845815f54b54219aaa5f8d8064ba050" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42867 } }
I20240501 07:59:00.952551 5366 raft_consensus.cc:374] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.952608 5366 raft_consensus.cc:729] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd24435aa8644984805bdbf02803ddd8, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.952687 5365 raft_consensus.cc:348] T a6c16f2cb2074e02acb65dcde42c0a09 P 12da6fa02ac940bea357b120f3d1029a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12da6fa02ac940bea357b120f3d1029a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38837 } } peers { permanent_uuid: "cd24435aa8644984805bdbf02803ddd8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37633 } } peers { permanent_uuid: "3845815f54b54219aaa5f8d8064ba050" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42867 } }
I20240501 07:59:00.952883 5365 raft_consensus.cc:374] T a6c16f2cb2074e02acb65dcde42c0a09 P 12da6fa02ac940bea357b120f3d1029a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.952940 5365 raft_consensus.cc:729] T a6c16f2cb2074e02acb65dcde42c0a09 P 12da6fa02ac940bea357b120f3d1029a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12da6fa02ac940bea357b120f3d1029a, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.953063 5366 consensus_queue.cc:260] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12da6fa02ac940bea357b120f3d1029a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38837 } } peers { permanent_uuid: "cd24435aa8644984805bdbf02803ddd8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37633 } } peers { permanent_uuid: "3845815f54b54219aaa5f8d8064ba050" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42867 } }
I20240501 07:59:00.953359 5365 consensus_queue.cc:260] T a6c16f2cb2074e02acb65dcde42c0a09 P 12da6fa02ac940bea357b120f3d1029a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12da6fa02ac940bea357b120f3d1029a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38837 } } peers { permanent_uuid: "cd24435aa8644984805bdbf02803ddd8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37633 } } peers { permanent_uuid: "3845815f54b54219aaa5f8d8064ba050" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42867 } }
I20240501 07:59:00.953737 5347 heartbeater.cc:503] Master 127.0.99.190:37191 was elected leader, sending a full tablet report...
I20240501 07:59:00.954005 5366 ts_tablet_manager.cc:1398] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 07:59:00.954037 5365 ts_tablet_manager.cc:1398] T a6c16f2cb2074e02acb65dcde42c0a09 P 12da6fa02ac940bea357b120f3d1029a: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240501 07:59:00.954360 4954 heartbeater.cc:503] Master 127.0.99.190:37191 was elected leader, sending a full tablet report...
W20240501 07:59:01.021764 5086 tablet.cc:2369] T a6c16f2cb2074e02acb65dcde42c0a09 P 3845815f54b54219aaa5f8d8064ba050: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:01.049682 5371 raft_consensus.cc:482] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:01.049890 5371 raft_consensus.cc:504] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12da6fa02ac940bea357b120f3d1029a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38837 } } peers { permanent_uuid: "cd24435aa8644984805bdbf02803ddd8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37633 } } peers { permanent_uuid: "3845815f54b54219aaa5f8d8064ba050" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42867 } }
I20240501 07:59:01.051002 5371 leader_election.cc:288] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12da6fa02ac940bea357b120f3d1029a (127.0.99.130:38837), 3845815f54b54219aaa5f8d8064ba050 (127.0.99.131:42867)
I20240501 07:59:01.055172 4908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c16f2cb2074e02acb65dcde42c0a09" candidate_uuid: "cd24435aa8644984805bdbf02803ddd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12da6fa02ac940bea357b120f3d1029a" is_pre_election: true
I20240501 07:59:01.055387 5039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c16f2cb2074e02acb65dcde42c0a09" candidate_uuid: "cd24435aa8644984805bdbf02803ddd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3845815f54b54219aaa5f8d8064ba050" is_pre_election: true
I20240501 07:59:01.055480 4908 raft_consensus.cc:2454] T a6c16f2cb2074e02acb65dcde42c0a09 P 12da6fa02ac940bea357b120f3d1029a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd24435aa8644984805bdbf02803ddd8 in term 0.
I20240501 07:59:01.055632 5039 raft_consensus.cc:2454] T a6c16f2cb2074e02acb65dcde42c0a09 P 3845815f54b54219aaa5f8d8064ba050 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd24435aa8644984805bdbf02803ddd8 in term 0.
I20240501 07:59:01.055938 5234 leader_election.cc:302] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12da6fa02ac940bea357b120f3d1029a, cd24435aa8644984805bdbf02803ddd8; no voters:
I20240501 07:59:01.056185 5371 raft_consensus.cc:2781] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:01.056255 5371 raft_consensus.cc:482] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:01.056284 5371 raft_consensus.cc:3037] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.057176 5371 raft_consensus.cc:504] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12da6fa02ac940bea357b120f3d1029a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38837 } } peers { permanent_uuid: "cd24435aa8644984805bdbf02803ddd8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37633 } } peers { permanent_uuid: "3845815f54b54219aaa5f8d8064ba050" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42867 } }
I20240501 07:59:01.057535 5371 leader_election.cc:288] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 [CANDIDATE]: Term 1 election: Requested vote from peers 12da6fa02ac940bea357b120f3d1029a (127.0.99.130:38837), 3845815f54b54219aaa5f8d8064ba050 (127.0.99.131:42867)
I20240501 07:59:01.057960 5039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c16f2cb2074e02acb65dcde42c0a09" candidate_uuid: "cd24435aa8644984805bdbf02803ddd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3845815f54b54219aaa5f8d8064ba050"
I20240501 07:59:01.057960 4908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c16f2cb2074e02acb65dcde42c0a09" candidate_uuid: "cd24435aa8644984805bdbf02803ddd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12da6fa02ac940bea357b120f3d1029a"
I20240501 07:59:01.058099 5039 raft_consensus.cc:3037] T a6c16f2cb2074e02acb65dcde42c0a09 P 3845815f54b54219aaa5f8d8064ba050 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.058130 4908 raft_consensus.cc:3037] T a6c16f2cb2074e02acb65dcde42c0a09 P 12da6fa02ac940bea357b120f3d1029a [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.059259 5039 raft_consensus.cc:2454] T a6c16f2cb2074e02acb65dcde42c0a09 P 3845815f54b54219aaa5f8d8064ba050 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd24435aa8644984805bdbf02803ddd8 in term 1.
I20240501 07:59:01.059259 4908 raft_consensus.cc:2454] T a6c16f2cb2074e02acb65dcde42c0a09 P 12da6fa02ac940bea357b120f3d1029a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd24435aa8644984805bdbf02803ddd8 in term 1.
I20240501 07:59:01.059599 5234 leader_election.cc:302] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12da6fa02ac940bea357b120f3d1029a, cd24435aa8644984805bdbf02803ddd8; no voters:
I20240501 07:59:01.059790 5371 raft_consensus.cc:2781] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:01.060026 5371 raft_consensus.cc:686] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 [term 1 LEADER]: Becoming Leader. State: Replica: cd24435aa8644984805bdbf02803ddd8, State: Running, Role: LEADER
I20240501 07:59:01.060348 5371 consensus_queue.cc:237] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12da6fa02ac940bea357b120f3d1029a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38837 } } peers { permanent_uuid: "cd24435aa8644984805bdbf02803ddd8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37633 } } peers { permanent_uuid: "3845815f54b54219aaa5f8d8064ba050" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42867 } }
I20240501 07:59:01.063277 4634 catalog_manager.cc:5484] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd24435aa8644984805bdbf02803ddd8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "cd24435aa8644984805bdbf02803ddd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12da6fa02ac940bea357b120f3d1029a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38837 } } peers { permanent_uuid: "cd24435aa8644984805bdbf02803ddd8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37633 } } peers { permanent_uuid: "3845815f54b54219aaa5f8d8064ba050" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42867 } } }
W20240501 07:59:01.077025 4955 tablet.cc:2369] T a6c16f2cb2074e02acb65dcde42c0a09 P 12da6fa02ac940bea357b120f3d1029a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:01.083997 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 07:59:01.086071 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f886adf6ee224ebfb3c783d93ef386e1 to finish bootstrapping
I20240501 07:59:01.093870 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 12da6fa02ac940bea357b120f3d1029a to finish bootstrapping
I20240501 07:59:01.099575 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3845815f54b54219aaa5f8d8064ba050 to finish bootstrapping
I20240501 07:59:01.104902 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4d38ff60089547989a10843c41f76a03 to finish bootstrapping
I20240501 07:59:01.111344 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver cd24435aa8644984805bdbf02803ddd8 to finish bootstrapping
W20240501 07:59:01.137080 5348 tablet.cc:2369] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:01.156054 4823 heartbeater.cc:503] Master 127.0.99.190:37191 was elected leader, sending a full tablet report...
I20240501 07:59:01.320631 5019 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.321043 4888 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.321056 5281 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:01.321727 5150 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.343521 4757 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.427325 5019 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.428431 5150 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.430091 5281 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:01.434769 4888 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.441670 4757 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.572403 5446 consensus_queue.cc:1035] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3845815f54b54219aaa5f8d8064ba050" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 42867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.579668 5446 consensus_queue.cc:1035] T a6c16f2cb2074e02acb65dcde42c0a09 P cd24435aa8644984805bdbf02803ddd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12da6fa02ac940bea357b120f3d1029a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 38837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.669209 4757 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.669481 5150 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.671001 4888 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.671361 5281 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:01.684947 5019 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.707665 5216 heartbeater.cc:503] Master 127.0.99.190:37191 was elected leader, sending a full tablet report...
I20240501 07:59:01.769678 5150 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.770574 4757 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.771361 4888 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.771620 5019 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.772492 5281 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:01.893765 5019 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.895073 4888 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.902498 4757 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.903713 5150 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:01.908419 5281 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:01.944020 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 4695
I20240501 07:59:01.949456 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 4826
I20240501 07:59:01.954843 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 4957
I20240501 07:59:01.960327 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 5088
I20240501 07:59:01.965206 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 5219
I20240501 07:59:01.970503 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 4604
2024-05-01T07:59:01Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (2264 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2264 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/2
I20240501 07:59:01.992238 398 test_util.cc:268] Using random seed: -602570968
I20240501 07:59:01.993158 398 ts_itest-base.cc:112] Starting cluster with:
I20240501 07:59:01.993203 398 ts_itest-base.cc:113] --------------
I20240501 07:59:01.993217 398 ts_itest-base.cc:114] 7 tablet servers
I20240501 07:59:01.993232 398 ts_itest-base.cc:115] 2 replicas per TS
I20240501 07:59:01.993244 398 ts_itest-base.cc:116] --------------
2024-05-01T07:59:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T07:59:01Z Disabled control of system clock
I20240501 07:59:02.001214 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:40255
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:40255
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 07:59:02.112182 5558 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 07:59:02.112471 5558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:02.112497 5558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:02.112542 5558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:02.115890 5558 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:02.115995 5558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 07:59:02.116019 5558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:02.116036 5558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 07:59:02.116053 5558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 07:59:02.120168 5558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:40255
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:40255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.5558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:02.121227 5558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:02.122483 5558 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:02.130337 5564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:02.130740 5566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:02.131332 5563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:02.131814 5558 server_base.cc:1034] running on GCE node
I20240501 07:59:02.132355 5558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:02.133366 5558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:02.134505 5558 hybrid_clock.cc:648] HybridClock initialized: now 1714550342134487 us; error 51 us; skew 500 ppm
I20240501 07:59:02.136533 5558 webserver.cc:460] Webserver started at http://127.0.99.190:46787/ using document root <none> and password file <none>
I20240501 07:59:02.137104 5558 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:02.137161 5558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:02.137341 5558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:02.139115 5558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6ba367f2ecb14c299239fdf04770cb81"
format_stamp: "Formatted at 2024-05-01 07:59:02 on dist-test-slave-2srn"
I20240501 07:59:02.139592 5558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6ba367f2ecb14c299239fdf04770cb81"
format_stamp: "Formatted at 2024-05-01 07:59:02 on dist-test-slave-2srn"
I20240501 07:59:02.143105 5558 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:59:02.145486 5572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:02.146595 5558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:02.146735 5558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6ba367f2ecb14c299239fdf04770cb81"
format_stamp: "Formatted at 2024-05-01 07:59:02 on dist-test-slave-2srn"
I20240501 07:59:02.146835 5558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:02.163229 5558 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:02.163954 5558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:02.164124 5558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:02.170773 5558 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:40255
I20240501 07:59:02.170799 5624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:40255 every 8 connection(s)
I20240501 07:59:02.171898 5558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 07:59:02.174592 5625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.177258 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 5558
I20240501 07:59:02.177395 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 07:59:02.180518 5625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81: Bootstrap starting.
I20240501 07:59:02.182740 5625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.183599 5625 log.cc:824] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:02.185858 5625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81: No bootstrap required, opened a new log
I20240501 07:59:02.188598 5625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba367f2ecb14c299239fdf04770cb81" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 40255 } }
I20240501 07:59:02.188832 5625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.188891 5625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba367f2ecb14c299239fdf04770cb81, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.189299 5625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba367f2ecb14c299239fdf04770cb81" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 40255 } }
I20240501 07:59:02.189533 5625 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:59:02.189627 5625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:59:02.189749 5625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.190668 5625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba367f2ecb14c299239fdf04770cb81" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 40255 } }
I20240501 07:59:02.191004 5625 leader_election.cc:302] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ba367f2ecb14c299239fdf04770cb81; no voters:
I20240501 07:59:02.191279 5625 leader_election.cc:288] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:59:02.191393 5630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:02.192034 5630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81 [term 1 LEADER]: Becoming Leader. State: Replica: 6ba367f2ecb14c299239fdf04770cb81, State: Running, Role: LEADER
I20240501 07:59:02.192404 5625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:59:02.192421 5630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba367f2ecb14c299239fdf04770cb81" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 40255 } }
I20240501 07:59:02.194149 5632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ba367f2ecb14c299239fdf04770cb81. Latest consensus state: current_term: 1 leader_uuid: "6ba367f2ecb14c299239fdf04770cb81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba367f2ecb14c299239fdf04770cb81" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 40255 } } }
I20240501 07:59:02.194284 5632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81 [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:02.194535 5631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ba367f2ecb14c299239fdf04770cb81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba367f2ecb14c299239fdf04770cb81" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 40255 } } }
I20240501 07:59:02.194626 5645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:59:02.194826 5631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81 [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:02.197623 5645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:59:02.203199 5645 catalog_manager.cc:1347] Generated new cluster ID: 6d0b0b7ceb4d4681ae99a31117365c0f
I20240501 07:59:02.203296 5645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:59:02.228730 5645 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:59:02.229691 5645 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:59:02.239141 5645 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6ba367f2ecb14c299239fdf04770cb81: Generated new TSK 0
I20240501 07:59:02.239948 5645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:59:02.245286 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40255
--builtin_ntp_servers=127.0.99.148:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 07:59:02.359326 5649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:02.359567 5649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:02.359611 5649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:02.362978 5649 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:02.363068 5649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:02.363143 5649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240501 07:59:02.367120 5649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40255
--never_fsync=true
--heap_profile_path=/tmp/kudu.5649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:02.368135 5649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:02.369364 5649 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:02.378204 5654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:02.378556 5655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:02.379169 5657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:02.379904 5649 server_base.cc:1034] running on GCE node
I20240501 07:59:02.380354 5649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:02.380942 5649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:02.382092 5649 hybrid_clock.cc:648] HybridClock initialized: now 1714550342382085 us; error 44 us; skew 500 ppm
I20240501 07:59:02.384156 5649 webserver.cc:460] Webserver started at http://127.0.99.129:37659/ using document root <none> and password file <none>
I20240501 07:59:02.384732 5649 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:02.384807 5649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:02.384989 5649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:02.386740 5649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "46c782f15fb643acb2ef72105f1fb04a"
format_stamp: "Formatted at 2024-05-01 07:59:02 on dist-test-slave-2srn"
I20240501 07:59:02.387220 5649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "46c782f15fb643acb2ef72105f1fb04a"
format_stamp: "Formatted at 2024-05-01 07:59:02 on dist-test-slave-2srn"
I20240501 07:59:02.390662 5649 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 07:59:02.392932 5663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:02.394222 5649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:02.394361 5649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "46c782f15fb643acb2ef72105f1fb04a"
format_stamp: "Formatted at 2024-05-01 07:59:02 on dist-test-slave-2srn"
I20240501 07:59:02.394495 5649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:02.412663 5649 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:02.413391 5649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:02.413561 5649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:02.414137 5649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:02.415112 5649 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:02.415165 5649 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:02.415221 5649 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:02.415248 5649 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:02.424178 5649 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:33797
I20240501 07:59:02.424192 5776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:33797 every 8 connection(s)
I20240501 07:59:02.425287 5649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 07:59:02.431733 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 5649
I20240501 07:59:02.431855 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 07:59:02.434260 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40255
--builtin_ntp_servers=127.0.99.148:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:02.434990 5777 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40255
I20240501 07:59:02.435252 5777 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:02.435736 5777 heartbeater.cc:511] Master 127.0.99.190:40255 requested a full tablet report, sending...
I20240501 07:59:02.437389 5589 ts_manager.cc:194] Registered new tserver with Master: 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797)
I20240501 07:59:02.438750 5589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:60949
W20240501 07:59:02.550382 5780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:02.550678 5780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:02.550745 5780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:02.554301 5780 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:02.554409 5780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:02.554550 5780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240501 07:59:02.558624 5780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40255
--never_fsync=true
--heap_profile_path=/tmp/kudu.5780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:02.559631 5780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:02.560873 5780 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:02.569900 5785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:02.570282 5786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:02.570904 5788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:02.571461 5780 server_base.cc:1034] running on GCE node
I20240501 07:59:02.571933 5780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:02.572978 5780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:02.574210 5780 hybrid_clock.cc:648] HybridClock initialized: now 1714550342574167 us; error 84 us; skew 500 ppm
I20240501 07:59:02.576460 5780 webserver.cc:460] Webserver started at http://127.0.99.130:40859/ using document root <none> and password file <none>
I20240501 07:59:02.577061 5780 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:02.577126 5780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:02.577309 5780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:02.579149 5780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f022e3632e9a428882a6cd16dc398723"
format_stamp: "Formatted at 2024-05-01 07:59:02 on dist-test-slave-2srn"
I20240501 07:59:02.579659 5780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f022e3632e9a428882a6cd16dc398723"
format_stamp: "Formatted at 2024-05-01 07:59:02 on dist-test-slave-2srn"
I20240501 07:59:02.583356 5780 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:59:02.586130 5794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:02.587491 5780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 07:59:02.587633 5780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f022e3632e9a428882a6cd16dc398723"
format_stamp: "Formatted at 2024-05-01 07:59:02 on dist-test-slave-2srn"
I20240501 07:59:02.587746 5780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:02.618960 5780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:02.619695 5780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:02.619861 5780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:02.620427 5780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:02.621388 5780 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:02.621438 5780 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:02.621492 5780 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:02.621520 5780 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:02.630153 5780 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:42187
I20240501 07:59:02.630234 5907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:42187 every 8 connection(s)
I20240501 07:59:02.631250 5780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 07:59:02.631618 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 5780
I20240501 07:59:02.631744 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 07:59:02.634635 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40255
--builtin_ntp_servers=127.0.99.148:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:02.641587 5908 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40255
I20240501 07:59:02.641837 5908 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:02.642383 5908 heartbeater.cc:511] Master 127.0.99.190:40255 requested a full tablet report, sending...
I20240501 07:59:02.643501 5589 ts_manager.cc:194] Registered new tserver with Master: f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187)
I20240501 07:59:02.644079 5589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:51241
W20240501 07:59:02.751330 5911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:02.751592 5911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:02.751657 5911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:02.754956 5911 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:02.755043 5911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:02.755122 5911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240501 07:59:02.760308 5911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40255
--never_fsync=true
--heap_profile_path=/tmp/kudu.5911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:02.761337 5911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:02.762542 5911 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:02.771477 5919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:02.771824 5916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:02.772424 5917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:02.772986 5911 server_base.cc:1034] running on GCE node
I20240501 07:59:02.773502 5911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:02.774266 5911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:02.775411 5911 hybrid_clock.cc:648] HybridClock initialized: now 1714550342775390 us; error 43 us; skew 500 ppm
I20240501 07:59:02.777505 5911 webserver.cc:460] Webserver started at http://127.0.99.131:34757/ using document root <none> and password file <none>
I20240501 07:59:02.778091 5911 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:02.778151 5911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:02.778345 5911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:02.780174 5911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ec372f1e72b040e780e6bdd4d5409d33"
format_stamp: "Formatted at 2024-05-01 07:59:02 on dist-test-slave-2srn"
I20240501 07:59:02.780696 5911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ec372f1e72b040e780e6bdd4d5409d33"
format_stamp: "Formatted at 2024-05-01 07:59:02 on dist-test-slave-2srn"
I20240501 07:59:02.784401 5911 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 07:59:02.787357 5925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:02.788661 5911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:59:02.788803 5911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ec372f1e72b040e780e6bdd4d5409d33"
format_stamp: "Formatted at 2024-05-01 07:59:02 on dist-test-slave-2srn"
I20240501 07:59:02.788916 5911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:02.802978 5911 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:02.803678 5911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:02.803850 5911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:02.804416 5911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:02.805371 5911 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:02.805421 5911 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:02.805476 5911 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:02.805505 5911 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:02.814889 5911 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:38531
I20240501 07:59:02.814952 6038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:38531 every 8 connection(s)
I20240501 07:59:02.815866 5911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 07:59:02.821031 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 5911
I20240501 07:59:02.821158 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 07:59:02.823343 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40255
--builtin_ntp_servers=127.0.99.148:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:02.827132 6039 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40255
I20240501 07:59:02.827397 6039 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:02.827916 6039 heartbeater.cc:511] Master 127.0.99.190:40255 requested a full tablet report, sending...
I20240501 07:59:02.829205 5589 ts_manager.cc:194] Registered new tserver with Master: ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531)
I20240501 07:59:02.829790 5589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:58881
W20240501 07:59:02.933262 6042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:02.933507 6042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:02.933552 6042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:02.936919 6042 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:02.937004 6042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:02.937074 6042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:59:02.941061 6042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40255
--never_fsync=true
--heap_profile_path=/tmp/kudu.6042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:02.942035 6042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:02.943217 6042 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:02.951822 6050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:02.952391 6047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:02.953092 6048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:02.953567 6042 server_base.cc:1034] running on GCE node
I20240501 07:59:02.954030 6042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:02.954715 6042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:02.955916 6042 hybrid_clock.cc:648] HybridClock initialized: now 1714550342955883 us; error 58 us; skew 500 ppm
I20240501 07:59:02.958209 6042 webserver.cc:460] Webserver started at http://127.0.99.132:33881/ using document root <none> and password file <none>
I20240501 07:59:02.958853 6042 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:02.958916 6042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:02.959097 6042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:02.960834 6042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "12f3f54871994efeb833126452c61da9"
format_stamp: "Formatted at 2024-05-01 07:59:02 on dist-test-slave-2srn"
I20240501 07:59:02.961333 6042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "12f3f54871994efeb833126452c61da9"
format_stamp: "Formatted at 2024-05-01 07:59:02 on dist-test-slave-2srn"
I20240501 07:59:02.964956 6042 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 07:59:02.967453 6056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:02.968750 6042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:02.968894 6042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "12f3f54871994efeb833126452c61da9"
format_stamp: "Formatted at 2024-05-01 07:59:02 on dist-test-slave-2srn"
I20240501 07:59:02.968997 6042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:02.982744 6042 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:02.983464 6042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:02.983630 6042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:02.984205 6042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:02.985174 6042 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:02.985229 6042 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:02.985285 6042 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:02.985311 6042 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:02.994225 6042 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:38041
I20240501 07:59:02.994262 6169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:38041 every 8 connection(s)
I20240501 07:59:02.995194 6042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:59:03.000022 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 6042
I20240501 07:59:03.000154 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 07:59:03.002502 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40255
--builtin_ntp_servers=127.0.99.148:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:03.004622 6170 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40255
I20240501 07:59:03.004894 6170 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:03.005393 6170 heartbeater.cc:511] Master 127.0.99.190:40255 requested a full tablet report, sending...
I20240501 07:59:03.006850 5589 ts_manager.cc:194] Registered new tserver with Master: 12f3f54871994efeb833126452c61da9 (127.0.99.132:38041)
I20240501 07:59:03.007658 5589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:54077
W20240501 07:59:03.114570 6173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:03.114820 6173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:03.114866 6173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:03.118031 6173 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:03.118111 6173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:03.118187 6173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:59:03.122000 6173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40255
--never_fsync=true
--heap_profile_path=/tmp/kudu.6173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:03.122987 6173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:03.124089 6173 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:03.132690 6178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:03.133070 6179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:03.133644 6181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:03.134191 6173 server_base.cc:1034] running on GCE node
I20240501 07:59:03.134630 6173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:03.135224 6173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:03.136381 6173 hybrid_clock.cc:648] HybridClock initialized: now 1714550343136370 us; error 38 us; skew 500 ppm
I20240501 07:59:03.142833 6173 webserver.cc:460] Webserver started at http://127.0.99.133:44059/ using document root <none> and password file <none>
I20240501 07:59:03.143463 6173 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:03.143529 6173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:03.143716 6173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:03.145498 6173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "72a601327d9d447aa7957cde6c36e000"
format_stamp: "Formatted at 2024-05-01 07:59:03 on dist-test-slave-2srn"
I20240501 07:59:03.145992 6173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "72a601327d9d447aa7957cde6c36e000"
format_stamp: "Formatted at 2024-05-01 07:59:03 on dist-test-slave-2srn"
I20240501 07:59:03.149520 6173 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240501 07:59:03.152012 6187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:03.153165 6173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:03.153302 6173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "72a601327d9d447aa7957cde6c36e000"
format_stamp: "Formatted at 2024-05-01 07:59:03 on dist-test-slave-2srn"
I20240501 07:59:03.153404 6173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:03.174068 6173 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:03.174803 6173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:03.174965 6173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:03.175513 6173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:03.176540 6173 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:03.176604 6173 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:03.176656 6173 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:03.176687 6173 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:03.185756 6173 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:34267
I20240501 07:59:03.185771 6300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:34267 every 8 connection(s)
I20240501 07:59:03.186776 6173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:59:03.189175 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 6173
I20240501 07:59:03.189293 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 07:59:03.191637 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40255
--builtin_ntp_servers=127.0.99.148:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:03.197083 6301 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40255
I20240501 07:59:03.197366 6301 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:03.198009 6301 heartbeater.cc:511] Master 127.0.99.190:40255 requested a full tablet report, sending...
I20240501 07:59:03.199486 5589 ts_manager.cc:194] Registered new tserver with Master: 72a601327d9d447aa7957cde6c36e000 (127.0.99.133:34267)
I20240501 07:59:03.200191 5589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:60247
W20240501 07:59:03.310515 6304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:03.310762 6304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:03.310808 6304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:03.314705 6304 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:03.314833 6304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:03.314916 6304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 07:59:03.319887 6304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40255
--never_fsync=true
--heap_profile_path=/tmp/kudu.6304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:03.320925 6304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:03.322135 6304 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:03.330781 6312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:03.331266 6309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:03.331748 6310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:03.332311 6304 server_base.cc:1034] running on GCE node
I20240501 07:59:03.332754 6304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:03.333350 6304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:03.334506 6304 hybrid_clock.cc:648] HybridClock initialized: now 1714550343334489 us; error 33 us; skew 500 ppm
I20240501 07:59:03.336550 6304 webserver.cc:460] Webserver started at http://127.0.99.134:36825/ using document root <none> and password file <none>
I20240501 07:59:03.337139 6304 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:03.337198 6304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:03.337378 6304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:03.339174 6304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "afc8c07c29354a04926027f57b8c6fab"
format_stamp: "Formatted at 2024-05-01 07:59:03 on dist-test-slave-2srn"
I20240501 07:59:03.339672 6304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "afc8c07c29354a04926027f57b8c6fab"
format_stamp: "Formatted at 2024-05-01 07:59:03 on dist-test-slave-2srn"
I20240501 07:59:03.343151 6304 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:59:03.345499 6318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:03.346691 6304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:03.346812 6304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "afc8c07c29354a04926027f57b8c6fab"
format_stamp: "Formatted at 2024-05-01 07:59:03 on dist-test-slave-2srn"
I20240501 07:59:03.346908 6304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:03.367759 6304 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:03.368558 6304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:03.368728 6304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:03.369319 6304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:03.370290 6304 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:03.370344 6304 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:03.370405 6304 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:03.370461 6304 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:03.379330 6304 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:38017
I20240501 07:59:03.379351 6431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:38017 every 8 connection(s)
I20240501 07:59:03.380312 6304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 07:59:03.380784 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 6304
I20240501 07:59:03.380887 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 07:59:03.383356 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40255
--builtin_ntp_servers=127.0.99.148:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:03.390606 6432 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40255
I20240501 07:59:03.390863 6432 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:03.391363 6432 heartbeater.cc:511] Master 127.0.99.190:40255 requested a full tablet report, sending...
I20240501 07:59:03.392514 5589 ts_manager.cc:194] Registered new tserver with Master: afc8c07c29354a04926027f57b8c6fab (127.0.99.134:38017)
I20240501 07:59:03.393191 5589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:46569
I20240501 07:59:03.441748 5777 heartbeater.cc:503] Master 127.0.99.190:40255 was elected leader, sending a full tablet report...
W20240501 07:59:03.497323 6435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:03.497579 6435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:03.497628 6435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:03.500942 6435 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:03.501039 6435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:03.501111 6435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 07:59:03.505051 6435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40255
--never_fsync=true
--heap_profile_path=/tmp/kudu.6435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:03.506004 6435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:03.507169 6435 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:03.515935 6441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:03.516265 6440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:03.516829 6443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:03.517438 6435 server_base.cc:1034] running on GCE node
I20240501 07:59:03.517912 6435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:03.518570 6435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:03.519765 6435 hybrid_clock.cc:648] HybridClock initialized: now 1714550343519725 us; error 58 us; skew 500 ppm
I20240501 07:59:03.521922 6435 webserver.cc:460] Webserver started at http://127.0.99.135:33295/ using document root <none> and password file <none>
I20240501 07:59:03.522579 6435 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:03.522660 6435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:03.522859 6435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:03.524616 6435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "a3af86b0e9a04ecbb15d305c48d7acb4"
format_stamp: "Formatted at 2024-05-01 07:59:03 on dist-test-slave-2srn"
I20240501 07:59:03.525141 6435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "a3af86b0e9a04ecbb15d305c48d7acb4"
format_stamp: "Formatted at 2024-05-01 07:59:03 on dist-test-slave-2srn"
I20240501 07:59:03.528695 6435 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 07:59:03.531234 6449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:03.532752 6435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:59:03.532899 6435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a3af86b0e9a04ecbb15d305c48d7acb4"
format_stamp: "Formatted at 2024-05-01 07:59:03 on dist-test-slave-2srn"
I20240501 07:59:03.533003 6435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:03.552300 6435 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:03.553002 6435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:03.553171 6435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:03.553726 6435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:03.554774 6435 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:03.554832 6435 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:03.554888 6435 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:03.554911 6435 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:03.563809 6435 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:45001
I20240501 07:59:03.563840 6562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:45001 every 8 connection(s)
I20240501 07:59:03.564852 6435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 07:59:03.570016 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 6435
I20240501 07:59:03.570139 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 07:59:03.574347 6563 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40255
I20240501 07:59:03.574637 6563 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:03.575237 6563 heartbeater.cc:511] Master 127.0.99.190:40255 requested a full tablet report, sending...
I20240501 07:59:03.576247 5589 ts_manager.cc:194] Registered new tserver with Master: a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135:45001)
I20240501 07:59:03.576817 5589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:54093
I20240501 07:59:03.585690 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 07:59:03.595571 5589 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40744:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 07:59:03.613831 6366 tablet_service.cc:1467] Processing CreateTablet for tablet ba935cdc94cf4eeab62cfee9e789e2eb (DEFAULT_TABLE table=TestTable [id=3d3b8576b2154d6784d2c6478d3bb28b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.613831 6497 tablet_service.cc:1467] Processing CreateTablet for tablet ba935cdc94cf4eeab62cfee9e789e2eb (DEFAULT_TABLE table=TestTable [id=3d3b8576b2154d6784d2c6478d3bb28b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.614851 6366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba935cdc94cf4eeab62cfee9e789e2eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.614851 6497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba935cdc94cf4eeab62cfee9e789e2eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.620477 6579 tablet_bootstrap.cc:492] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4: Bootstrap starting.
I20240501 07:59:03.620527 6580 tablet_bootstrap.cc:492] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab: Bootstrap starting.
I20240501 07:59:03.622690 6579 tablet_bootstrap.cc:654] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.622690 6580 tablet_bootstrap.cc:654] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.623386 6579 log.cc:824] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:03.623386 6580 log.cc:824] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:03.625245 6579 tablet_bootstrap.cc:492] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4: No bootstrap required, opened a new log
I20240501 07:59:03.625245 6580 tablet_bootstrap.cc:492] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab: No bootstrap required, opened a new log
I20240501 07:59:03.625456 6580 ts_tablet_manager.cc:1367] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 07:59:03.625456 6579 ts_tablet_manager.cc:1367] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 07:59:03.629421 6579 raft_consensus.cc:348] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:03.629421 6580 raft_consensus.cc:348] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:03.629689 6580 raft_consensus.cc:374] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.629689 6579 raft_consensus.cc:374] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.629740 6579 raft_consensus.cc:729] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3af86b0e9a04ecbb15d305c48d7acb4, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.629740 6580 raft_consensus.cc:729] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc8c07c29354a04926027f57b8c6fab, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.630280 6580 consensus_queue.cc:260] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:03.630854 6579 consensus_queue.cc:260] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:03.631248 6580 ts_tablet_manager.cc:1398] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240501 07:59:03.631601 6432 heartbeater.cc:503] Master 127.0.99.190:40255 was elected leader, sending a full tablet report...
I20240501 07:59:03.631773 6579 ts_tablet_manager.cc:1398] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 07:59:03.631814 6563 heartbeater.cc:503] Master 127.0.99.190:40255 was elected leader, sending a full tablet report...
I20240501 07:59:03.646945 5908 heartbeater.cc:503] Master 127.0.99.190:40255 was elected leader, sending a full tablet report...
W20240501 07:59:03.815752 6564 tablet.cc:2369] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:03.832219 6039 heartbeater.cc:503] Master 127.0.99.190:40255 was elected leader, sending a full tablet report...
W20240501 07:59:03.881589 6433 tablet.cc:2369] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:03.974522 6584 raft_consensus.cc:482] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:03.974730 6584 raft_consensus.cc:504] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:03.975764 6584 leader_election.cc:288] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afc8c07c29354a04926027f57b8c6fab (127.0.99.134:38017)
I20240501 07:59:03.979768 6385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba935cdc94cf4eeab62cfee9e789e2eb" candidate_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afc8c07c29354a04926027f57b8c6fab" is_pre_election: true
I20240501 07:59:03.980088 6385 raft_consensus.cc:2454] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3af86b0e9a04ecbb15d305c48d7acb4 in term 0.
I20240501 07:59:03.980619 6450 leader_election.cc:302] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a3af86b0e9a04ecbb15d305c48d7acb4, afc8c07c29354a04926027f57b8c6fab; no voters:
I20240501 07:59:03.981017 6584 raft_consensus.cc:2781] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:03.981092 6584 raft_consensus.cc:482] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:03.981120 6584 raft_consensus.cc:3037] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.981997 6584 raft_consensus.cc:504] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:03.982301 6584 leader_election.cc:288] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [CANDIDATE]: Term 1 election: Requested vote from peers afc8c07c29354a04926027f57b8c6fab (127.0.99.134:38017)
I20240501 07:59:03.982793 6385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba935cdc94cf4eeab62cfee9e789e2eb" candidate_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afc8c07c29354a04926027f57b8c6fab"
I20240501 07:59:03.982923 6385 raft_consensus.cc:3037] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.983784 6385 raft_consensus.cc:2454] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3af86b0e9a04ecbb15d305c48d7acb4 in term 1.
I20240501 07:59:03.984108 6450 leader_election.cc:302] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a3af86b0e9a04ecbb15d305c48d7acb4, afc8c07c29354a04926027f57b8c6fab; no voters:
I20240501 07:59:03.984311 6584 raft_consensus.cc:2781] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:03.984555 6584 raft_consensus.cc:686] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 LEADER]: Becoming Leader. State: Replica: a3af86b0e9a04ecbb15d305c48d7acb4, State: Running, Role: LEADER
I20240501 07:59:03.984874 6584 consensus_queue.cc:237] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:03.987749 5588 catalog_manager.cc:5484] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:04.009970 6170 heartbeater.cc:503] Master 127.0.99.190:40255 was elected leader, sending a full tablet report...
I20240501 07:59:04.061668 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 07:59:04.063105 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 46c782f15fb643acb2ef72105f1fb04a to finish bootstrapping
I20240501 07:59:04.069427 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f022e3632e9a428882a6cd16dc398723 to finish bootstrapping
I20240501 07:59:04.075361 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ec372f1e72b040e780e6bdd4d5409d33 to finish bootstrapping
I20240501 07:59:04.081091 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 12f3f54871994efeb833126452c61da9 to finish bootstrapping
I20240501 07:59:04.086923 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 72a601327d9d447aa7957cde6c36e000 to finish bootstrapping
I20240501 07:59:04.092880 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver afc8c07c29354a04926027f57b8c6fab to finish bootstrapping
I20240501 07:59:04.098801 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a3af86b0e9a04ecbb15d305c48d7acb4 to finish bootstrapping
I20240501 07:59:04.104027 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 6042
I20240501 07:59:04.109021 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 6173
I20240501 07:59:04.113843 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 6304
I20240501 07:59:04.119072 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 6435
I20240501 07:59:07.876005 5588 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40744:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:07.889521 5841 tablet_service.cc:1467] Processing CreateTablet for tablet 65e9dbbdb1fa459fabc258f9e6878f00 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3a5d5337264d45beb86597f2d8ffe305]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.890031 5710 tablet_service.cc:1467] Processing CreateTablet for tablet 1e15c1939984497fb1795ef7502545ca (DEFAULT_TABLE table=rebalance_test_table_0 [id=3a5d5337264d45beb86597f2d8ffe305]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.890030 5711 tablet_service.cc:1467] Processing CreateTablet for tablet 114e9265cef4425799ad44e73e053b59 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3a5d5337264d45beb86597f2d8ffe305]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.890671 5841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65e9dbbdb1fa459fabc258f9e6878f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.890969 5710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e15c1939984497fb1795ef7502545ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.891331 5711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 114e9265cef4425799ad44e73e053b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.891530 5973 tablet_service.cc:1467] Processing CreateTablet for tablet 1e15c1939984497fb1795ef7502545ca (DEFAULT_TABLE table=rebalance_test_table_0 [id=3a5d5337264d45beb86597f2d8ffe305]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.892446 5973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e15c1939984497fb1795ef7502545ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.893044 5972 tablet_service.cc:1467] Processing CreateTablet for tablet 65e9dbbdb1fa459fabc258f9e6878f00 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3a5d5337264d45beb86597f2d8ffe305]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.893329 5972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65e9dbbdb1fa459fabc258f9e6878f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.897308 6603 tablet_bootstrap.cc:492] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a: Bootstrap starting.
I20240501 07:59:07.899705 6603 tablet_bootstrap.cc:654] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.889532 5842 tablet_service.cc:1467] Processing CreateTablet for tablet 114e9265cef4425799ad44e73e053b59 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3a5d5337264d45beb86597f2d8ffe305]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.900264 6605 tablet_bootstrap.cc:492] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33: Bootstrap starting.
I20240501 07:59:07.900419 5842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 114e9265cef4425799ad44e73e053b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.900545 6603 log.cc:824] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:07.902218 6605 tablet_bootstrap.cc:654] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.902457 6603 tablet_bootstrap.cc:492] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a: No bootstrap required, opened a new log
I20240501 07:59:07.902637 6603 ts_tablet_manager.cc:1367] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240501 07:59:07.903069 6604 tablet_bootstrap.cc:492] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723: Bootstrap starting.
I20240501 07:59:07.903105 6605 log.cc:824] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:07.904942 6605 tablet_bootstrap.cc:492] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33: No bootstrap required, opened a new log
I20240501 07:59:07.905151 6604 tablet_bootstrap.cc:654] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.905149 6605 ts_tablet_manager.cc:1367] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 07:59:07.905827 6603 raft_consensus.cc:348] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:07.905925 6604 log.cc:824] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:07.906013 6603 raft_consensus.cc:374] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.906085 6603 raft_consensus.cc:729] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c782f15fb643acb2ef72105f1fb04a, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.906569 6603 consensus_queue.cc:260] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:07.907465 6603 ts_tablet_manager.cc:1398] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:59:07.907727 6603 tablet_bootstrap.cc:492] T 1e15c1939984497fb1795ef7502545ca P 46c782f15fb643acb2ef72105f1fb04a: Bootstrap starting.
I20240501 07:59:07.908504 6603 tablet_bootstrap.cc:654] T 1e15c1939984497fb1795ef7502545ca P 46c782f15fb643acb2ef72105f1fb04a: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.908887 6605 raft_consensus.cc:348] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:07.909180 6605 raft_consensus.cc:374] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.909276 6605 raft_consensus.cc:729] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.909571 6603 tablet_bootstrap.cc:492] T 1e15c1939984497fb1795ef7502545ca P 46c782f15fb643acb2ef72105f1fb04a: No bootstrap required, opened a new log
I20240501 07:59:07.909655 6603 ts_tablet_manager.cc:1367] T 1e15c1939984497fb1795ef7502545ca P 46c782f15fb643acb2ef72105f1fb04a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:07.909811 6604 tablet_bootstrap.cc:492] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723: No bootstrap required, opened a new log
I20240501 07:59:07.909781 6605 consensus_queue.cc:260] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:07.909982 6604 ts_tablet_manager.cc:1367] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 07:59:07.910192 6603 raft_consensus.cc:348] T 1e15c1939984497fb1795ef7502545ca P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:07.910295 6603 raft_consensus.cc:374] T 1e15c1939984497fb1795ef7502545ca P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.910326 6603 raft_consensus.cc:729] T 1e15c1939984497fb1795ef7502545ca P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c782f15fb643acb2ef72105f1fb04a, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.910439 6603 consensus_queue.cc:260] T 1e15c1939984497fb1795ef7502545ca P 46c782f15fb643acb2ef72105f1fb04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:07.910677 6605 ts_tablet_manager.cc:1398] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20240501 07:59:07.910688 6603 ts_tablet_manager.cc:1398] T 1e15c1939984497fb1795ef7502545ca P 46c782f15fb643acb2ef72105f1fb04a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:07.911938 6605 tablet_bootstrap.cc:492] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33: Bootstrap starting.
I20240501 07:59:07.913291 6605 tablet_bootstrap.cc:654] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.913961 6604 raft_consensus.cc:348] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:07.914297 6604 raft_consensus.cc:374] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.914587 6604 raft_consensus.cc:729] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.915007 6605 tablet_bootstrap.cc:492] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33: No bootstrap required, opened a new log
I20240501 07:59:07.915100 6605 ts_tablet_manager.cc:1367] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:07.915150 6604 consensus_queue.cc:260] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:07.915673 6605 raft_consensus.cc:348] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:07.915781 6605 raft_consensus.cc:374] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.915828 6605 raft_consensus.cc:729] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.915951 6605 consensus_queue.cc:260] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:07.916112 6604 ts_tablet_manager.cc:1398] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20240501 07:59:07.916213 6605 ts_tablet_manager.cc:1398] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:07.916517 6604 tablet_bootstrap.cc:492] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723: Bootstrap starting.
I20240501 07:59:07.917570 6604 tablet_bootstrap.cc:654] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.918733 6604 tablet_bootstrap.cc:492] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723: No bootstrap required, opened a new log
I20240501 07:59:07.918819 6604 ts_tablet_manager.cc:1367] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 07:59:07.919309 6604 raft_consensus.cc:348] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:07.919417 6604 raft_consensus.cc:374] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.919450 6604 raft_consensus.cc:729] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.919572 6604 consensus_queue.cc:260] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:07.919816 6604 ts_tablet_manager.cc:1398] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240501 07:59:07.931007 5778 tablet.cc:2369] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:07.973229 6611 raft_consensus.cc:482] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:07.973433 6611 raft_consensus.cc:504] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:07.974493 6611 leader_election.cc:288] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797)
I20240501 07:59:07.978351 5731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e15c1939984497fb1795ef7502545ca" candidate_uuid: "ec372f1e72b040e780e6bdd4d5409d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c782f15fb643acb2ef72105f1fb04a" is_pre_election: true
I20240501 07:59:07.978724 5731 raft_consensus.cc:2454] T 1e15c1939984497fb1795ef7502545ca P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec372f1e72b040e780e6bdd4d5409d33 in term 0.
I20240501 07:59:07.979195 5928 leader_election.cc:302] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, ec372f1e72b040e780e6bdd4d5409d33; no voters:
I20240501 07:59:07.979458 6611 raft_consensus.cc:2781] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:07.979528 6611 raft_consensus.cc:482] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:07.979559 6611 raft_consensus.cc:3037] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.980448 6611 raft_consensus.cc:504] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:07.980758 6611 leader_election.cc:288] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 election: Requested vote from peers 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797)
I20240501 07:59:07.981297 5731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e15c1939984497fb1795ef7502545ca" candidate_uuid: "ec372f1e72b040e780e6bdd4d5409d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c782f15fb643acb2ef72105f1fb04a"
I20240501 07:59:07.981428 5731 raft_consensus.cc:3037] T 1e15c1939984497fb1795ef7502545ca P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.982326 5731 raft_consensus.cc:2454] T 1e15c1939984497fb1795ef7502545ca P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec372f1e72b040e780e6bdd4d5409d33 in term 1.
I20240501 07:59:07.982672 5928 leader_election.cc:302] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, ec372f1e72b040e780e6bdd4d5409d33; no voters:
I20240501 07:59:07.982858 6611 raft_consensus.cc:2781] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:07.983074 6611 raft_consensus.cc:686] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Becoming Leader. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Running, Role: LEADER
I20240501 07:59:07.983461 6611 consensus_queue.cc:237] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:07.986224 5588 catalog_manager.cc:5484] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } }
I20240501 07:59:08.002094 6611 raft_consensus.cc:482] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:08.002239 6611 raft_consensus.cc:504] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:08.002703 6611 leader_election.cc:288] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187)
I20240501 07:59:08.006201 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65e9dbbdb1fa459fabc258f9e6878f00" candidate_uuid: "ec372f1e72b040e780e6bdd4d5409d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f022e3632e9a428882a6cd16dc398723" is_pre_election: true
I20240501 07:59:08.006541 5862 raft_consensus.cc:2454] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec372f1e72b040e780e6bdd4d5409d33 in term 0.
I20240501 07:59:08.006945 5926 leader_election.cc:302] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ec372f1e72b040e780e6bdd4d5409d33, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:08.007174 6611 raft_consensus.cc:2781] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:08.007234 6611 raft_consensus.cc:482] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:08.007261 6611 raft_consensus.cc:3037] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.008033 6611 raft_consensus.cc:504] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:08.008378 6611 leader_election.cc:288] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 election: Requested vote from peers f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187)
I20240501 07:59:08.008885 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65e9dbbdb1fa459fabc258f9e6878f00" candidate_uuid: "ec372f1e72b040e780e6bdd4d5409d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f022e3632e9a428882a6cd16dc398723"
I20240501 07:59:08.009006 5862 raft_consensus.cc:3037] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.010011 5862 raft_consensus.cc:2454] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec372f1e72b040e780e6bdd4d5409d33 in term 1.
I20240501 07:59:08.010476 5926 leader_election.cc:302] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ec372f1e72b040e780e6bdd4d5409d33, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:08.010704 6611 raft_consensus.cc:2781] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:08.010802 6611 raft_consensus.cc:686] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Becoming Leader. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Running, Role: LEADER
I20240501 07:59:08.010955 6611 consensus_queue.cc:237] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:08.012503 5588 catalog_manager.cc:5484] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } }
W20240501 07:59:08.071607 6040 tablet.cc:2369] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:08.074925 6613 raft_consensus.cc:482] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:08.075066 6613 raft_consensus.cc:504] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.076032 6613 leader_election.cc:288] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797)
I20240501 07:59:08.080576 5731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114e9265cef4425799ad44e73e053b59" candidate_uuid: "f022e3632e9a428882a6cd16dc398723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c782f15fb643acb2ef72105f1fb04a" is_pre_election: true
I20240501 07:59:08.080741 5731 raft_consensus.cc:2454] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f022e3632e9a428882a6cd16dc398723 in term 0.
I20240501 07:59:08.081193 5797 leader_election.cc:302] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:08.081480 6613 raft_consensus.cc:2781] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:08.081557 6613 raft_consensus.cc:482] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:08.081601 6613 raft_consensus.cc:3037] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.082633 6613 raft_consensus.cc:504] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.082952 6613 leader_election.cc:288] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 election: Requested vote from peers 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797)
I20240501 07:59:08.083477 5731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114e9265cef4425799ad44e73e053b59" candidate_uuid: "f022e3632e9a428882a6cd16dc398723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c782f15fb643acb2ef72105f1fb04a"
I20240501 07:59:08.083595 5731 raft_consensus.cc:3037] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.084439 5731 raft_consensus.cc:2454] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f022e3632e9a428882a6cd16dc398723 in term 1.
I20240501 07:59:08.084781 5797 leader_election.cc:302] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:08.084993 6613 raft_consensus.cc:2781] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:08.085335 6613 raft_consensus.cc:686] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Becoming Leader. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Running, Role: LEADER
I20240501 07:59:08.085700 6613 consensus_queue.cc:237] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.091281 5588 catalog_manager.cc:5484] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 reported cstate change: term changed from 0 to 1, leader changed from <none> to f022e3632e9a428882a6cd16dc398723 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } }
W20240501 07:59:08.136869 5909 tablet.cc:2369] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:59:08.236191 6622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:08.236505 6622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:08.239790 6622 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 07:59:08.341922 5731 raft_consensus.cc:1261] T 1e15c1939984497fb1795ef7502545ca P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:08.342067 5862 raft_consensus.cc:1261] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:08.342690 6616 consensus_queue.cc:1035] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.342690 6611 consensus_queue.cc:1035] T 1e15c1939984497fb1795ef7502545ca P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.347884 6652 mvcc.cc:205] Tried to move back new op lower bound from 7022798226803531776 to 7022798225454129152. Current Snapshot: MvccSnapshot[applied={T|T < 7022798226803531776}]
I20240501 07:59:08.349570 5730 raft_consensus.cc:1261] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:08.349565 6654 mvcc.cc:205] Tried to move back new op lower bound from 7022798226803531776 to 7022798225454129152. Current Snapshot: MvccSnapshot[applied={T|T < 7022798226803531776}]
I20240501 07:59:08.350620 6613 consensus_queue.cc:1035] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.360986 6656 mvcc.cc:205] Tried to move back new op lower bound from 7022798226800988160 to 7022798225342480384. Current Snapshot: MvccSnapshot[applied={T|T < 7022798226800988160}]
INSERT report
rows total: 2000
time total: 102.911 ms
time per row: 0.0514556 ms
I20240501 07:59:08.394079 5588 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40744:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:08.399899 5842 tablet_service.cc:1467] Processing CreateTablet for tablet bd95a7eff75845c0b49790c9657ef1e0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b2c8da5392b24236b9cc0f5c750c0528]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.399997 5710 tablet_service.cc:1467] Processing CreateTablet for tablet bd95a7eff75845c0b49790c9657ef1e0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b2c8da5392b24236b9cc0f5c750c0528]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.400092 5711 tablet_service.cc:1467] Processing CreateTablet for tablet 3f187bcb4c494805a8d6f366732b695a (DEFAULT_TABLE table=rebalance_test_table_1 [id=b2c8da5392b24236b9cc0f5c750c0528]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.400280 5710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd95a7eff75845c0b49790c9657ef1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.400288 5842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd95a7eff75845c0b49790c9657ef1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.400553 5711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f187bcb4c494805a8d6f366732b695a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.400893 5972 tablet_service.cc:1467] Processing CreateTablet for tablet a07321bfc4c1445a906ec7c1261693e9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b2c8da5392b24236b9cc0f5c750c0528]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.401155 5972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a07321bfc4c1445a906ec7c1261693e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.402549 6603 tablet_bootstrap.cc:492] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a: Bootstrap starting.
I20240501 07:59:08.402799 5973 tablet_service.cc:1467] Processing CreateTablet for tablet 3f187bcb4c494805a8d6f366732b695a (DEFAULT_TABLE table=rebalance_test_table_1 [id=b2c8da5392b24236b9cc0f5c750c0528]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.403041 5973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f187bcb4c494805a8d6f366732b695a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.403156 6605 tablet_bootstrap.cc:492] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33: Bootstrap starting.
I20240501 07:59:08.403599 6603 tablet_bootstrap.cc:654] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.404104 6605 tablet_bootstrap.cc:654] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.405092 6603 tablet_bootstrap.cc:492] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a: No bootstrap required, opened a new log
I20240501 07:59:08.405174 6604 tablet_bootstrap.cc:492] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723: Bootstrap starting.
I20240501 07:59:08.405186 6603 ts_tablet_manager.cc:1367] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:59:08.405695 6603 raft_consensus.cc:348] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.405812 6603 raft_consensus.cc:374] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.405844 6603 raft_consensus.cc:729] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c782f15fb643acb2ef72105f1fb04a, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.405992 6603 consensus_queue.cc:260] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.406114 6604 tablet_bootstrap.cc:654] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.406334 6603 ts_tablet_manager.cc:1398] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:08.407125 5842 tablet_service.cc:1467] Processing CreateTablet for tablet a07321bfc4c1445a906ec7c1261693e9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b2c8da5392b24236b9cc0f5c750c0528]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.407200 6603 tablet_bootstrap.cc:492] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a: Bootstrap starting.
I20240501 07:59:08.407398 5842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a07321bfc4c1445a906ec7c1261693e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.408151 6603 tablet_bootstrap.cc:654] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.408443 6605 tablet_bootstrap.cc:492] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33: No bootstrap required, opened a new log
I20240501 07:59:08.408537 6605 ts_tablet_manager.cc:1367] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:59:08.409026 6605 raft_consensus.cc:348] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:08.409137 6605 raft_consensus.cc:374] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.409168 6605 raft_consensus.cc:729] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.409256 6604 tablet_bootstrap.cc:492] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723: No bootstrap required, opened a new log
I20240501 07:59:08.409341 6604 ts_tablet_manager.cc:1367] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:08.409310 6605 consensus_queue.cc:260] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:08.409593 6605 ts_tablet_manager.cc:1398] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.409777 6605 tablet_bootstrap.cc:492] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33: Bootstrap starting.
I20240501 07:59:08.409952 6604 raft_consensus.cc:348] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.410053 6604 raft_consensus.cc:374] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.410089 6604 raft_consensus.cc:729] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.410218 6604 consensus_queue.cc:260] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.410705 6604 ts_tablet_manager.cc:1398] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:08.410880 6604 tablet_bootstrap.cc:492] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723: Bootstrap starting.
I20240501 07:59:08.410985 6603 tablet_bootstrap.cc:492] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a: No bootstrap required, opened a new log
I20240501 07:59:08.411069 6603 ts_tablet_manager.cc:1367] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:08.411589 6605 tablet_bootstrap.cc:654] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.411557 6603 raft_consensus.cc:348] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.411660 6603 raft_consensus.cc:374] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.411706 6603 raft_consensus.cc:729] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c782f15fb643acb2ef72105f1fb04a, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.411844 6603 consensus_queue.cc:260] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.411926 6604 tablet_bootstrap.cc:654] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.412143 6603 ts_tablet_manager.cc:1398] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.413192 6605 tablet_bootstrap.cc:492] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33: No bootstrap required, opened a new log
I20240501 07:59:08.413465 6605 ts_tablet_manager.cc:1367] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:08.414501 6604 tablet_bootstrap.cc:492] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723: No bootstrap required, opened a new log
I20240501 07:59:08.414654 6605 raft_consensus.cc:348] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.414749 6604 ts_tablet_manager.cc:1367] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 07:59:08.414826 6605 raft_consensus.cc:374] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.414916 6605 raft_consensus.cc:729] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.415082 6605 consensus_queue.cc:260] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.415444 6605 ts_tablet_manager.cc:1398] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:08.415310 6604 raft_consensus.cc:348] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:08.415848 6604 raft_consensus.cc:374] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.415917 6604 raft_consensus.cc:729] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.416093 6604 consensus_queue.cc:260] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:08.416381 6604 ts_tablet_manager.cc:1398] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:08.442493 6613 raft_consensus.cc:482] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:08.442622 6613 raft_consensus.cc:504] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:08.443009 6613 leader_election.cc:288] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531)
I20240501 07:59:08.446300 5993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07321bfc4c1445a906ec7c1261693e9" candidate_uuid: "f022e3632e9a428882a6cd16dc398723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec372f1e72b040e780e6bdd4d5409d33" is_pre_election: true
I20240501 07:59:08.446564 5993 raft_consensus.cc:2454] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f022e3632e9a428882a6cd16dc398723 in term 0.
I20240501 07:59:08.446890 5797 leader_election.cc:302] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ec372f1e72b040e780e6bdd4d5409d33, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:08.447098 6613 raft_consensus.cc:2781] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:08.447162 6613 raft_consensus.cc:482] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:08.447199 6613 raft_consensus.cc:3037] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.448063 6613 raft_consensus.cc:504] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:08.448385 6613 leader_election.cc:288] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 election: Requested vote from peers ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531)
I20240501 07:59:08.448817 5993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07321bfc4c1445a906ec7c1261693e9" candidate_uuid: "f022e3632e9a428882a6cd16dc398723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec372f1e72b040e780e6bdd4d5409d33"
I20240501 07:59:08.448938 5993 raft_consensus.cc:3037] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.449857 5993 raft_consensus.cc:2454] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f022e3632e9a428882a6cd16dc398723 in term 1.
I20240501 07:59:08.450165 5797 leader_election.cc:302] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ec372f1e72b040e780e6bdd4d5409d33, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:08.450377 6613 raft_consensus.cc:2781] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:08.450506 6613 raft_consensus.cc:686] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Becoming Leader. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Running, Role: LEADER
I20240501 07:59:08.450665 6613 consensus_queue.cc:237] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:08.452193 5588 catalog_manager.cc:5484] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 reported cstate change: term changed from 0 to 1, leader changed from <none> to f022e3632e9a428882a6cd16dc398723 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } }
I20240501 07:59:08.513643 6611 raft_consensus.cc:482] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:08.513800 6611 raft_consensus.cc:504] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.514169 6611 leader_election.cc:288] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797)
I20240501 07:59:08.514732 5730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f187bcb4c494805a8d6f366732b695a" candidate_uuid: "ec372f1e72b040e780e6bdd4d5409d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c782f15fb643acb2ef72105f1fb04a" is_pre_election: true
I20240501 07:59:08.514897 5730 raft_consensus.cc:2454] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec372f1e72b040e780e6bdd4d5409d33 in term 0.
I20240501 07:59:08.515271 5928 leader_election.cc:302] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, ec372f1e72b040e780e6bdd4d5409d33; no voters:
I20240501 07:59:08.515429 6611 raft_consensus.cc:2781] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:08.515486 6611 raft_consensus.cc:482] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:08.515514 6611 raft_consensus.cc:3037] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.516275 6611 raft_consensus.cc:504] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.516579 6611 leader_election.cc:288] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 election: Requested vote from peers 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797)
I20240501 07:59:08.517030 5730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f187bcb4c494805a8d6f366732b695a" candidate_uuid: "ec372f1e72b040e780e6bdd4d5409d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c782f15fb643acb2ef72105f1fb04a"
I20240501 07:59:08.517215 5730 raft_consensus.cc:3037] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.518118 5730 raft_consensus.cc:2454] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec372f1e72b040e780e6bdd4d5409d33 in term 1.
I20240501 07:59:08.518522 5928 leader_election.cc:302] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, ec372f1e72b040e780e6bdd4d5409d33; no voters:
I20240501 07:59:08.518746 6611 raft_consensus.cc:2781] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:08.518837 6611 raft_consensus.cc:686] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Becoming Leader. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Running, Role: LEADER
I20240501 07:59:08.518975 6611 consensus_queue.cc:237] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.520536 5588 catalog_manager.cc:5484] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } }
I20240501 07:59:08.559147 6609 raft_consensus.cc:482] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:08.559290 6609 raft_consensus.cc:504] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.560250 6609 leader_election.cc:288] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187)
I20240501 07:59:08.564497 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd95a7eff75845c0b49790c9657ef1e0" candidate_uuid: "46c782f15fb643acb2ef72105f1fb04a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f022e3632e9a428882a6cd16dc398723" is_pre_election: true
I20240501 07:59:08.564667 5862 raft_consensus.cc:2454] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c782f15fb643acb2ef72105f1fb04a in term 0.
I20240501 07:59:08.565152 5664 leader_election.cc:302] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:08.565435 6609 raft_consensus.cc:2781] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:08.565500 6609 raft_consensus.cc:482] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:08.565526 6609 raft_consensus.cc:3037] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.566344 6609 raft_consensus.cc:504] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.566695 6609 leader_election.cc:288] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [CANDIDATE]: Term 1 election: Requested vote from peers f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187)
I20240501 07:59:08.567148 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd95a7eff75845c0b49790c9657ef1e0" candidate_uuid: "46c782f15fb643acb2ef72105f1fb04a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f022e3632e9a428882a6cd16dc398723"
I20240501 07:59:08.567270 5862 raft_consensus.cc:3037] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.568064 5862 raft_consensus.cc:2454] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c782f15fb643acb2ef72105f1fb04a in term 1.
I20240501 07:59:08.568400 5664 leader_election.cc:302] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:08.568601 6609 raft_consensus.cc:2781] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:08.568818 6609 raft_consensus.cc:686] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [term 1 LEADER]: Becoming Leader. State: Replica: 46c782f15fb643acb2ef72105f1fb04a, State: Running, Role: LEADER
I20240501 07:59:08.569139 6609 consensus_queue.cc:237] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.571146 5588 catalog_manager.cc:5484] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "46c782f15fb643acb2ef72105f1fb04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } }
W20240501 07:59:08.708432 6681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:08.708717 6681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:08.711905 6681 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 07:59:08.808492 5862 raft_consensus.cc:1261] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Refusing update from remote peer 46c782f15fb643acb2ef72105f1fb04a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:08.809417 6609 consensus_queue.cc:1035] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.812815 5993 raft_consensus.cc:1261] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:08.813854 6613 consensus_queue.cc:1035] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.815218 5730 raft_consensus.cc:1261] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:08.821166 6611 consensus_queue.cc:1035] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 114.653 ms
time per row: 0.0573266 ms
I20240501 07:59:08.870673 5583 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40744:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:08.876515 5842 tablet_service.cc:1467] Processing CreateTablet for tablet 662d37043f054aa1a562cde8d6c155a9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f3b4504f12d4447ba80f01cbb31b8c0a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.876528 5710 tablet_service.cc:1467] Processing CreateTablet for tablet 7f4b2166aae54490a36ced8d37caf7b1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f3b4504f12d4447ba80f01cbb31b8c0a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.876557 5711 tablet_service.cc:1467] Processing CreateTablet for tablet 662d37043f054aa1a562cde8d6c155a9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f3b4504f12d4447ba80f01cbb31b8c0a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.876853 5842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 662d37043f054aa1a562cde8d6c155a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.876857 5710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f4b2166aae54490a36ced8d37caf7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.876998 5711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 662d37043f054aa1a562cde8d6c155a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.877023 5841 tablet_service.cc:1467] Processing CreateTablet for tablet d26c0973a2dc4c639bc3a616e28fbfef (DEFAULT_TABLE table=rebalance_test_table_2 [id=f3b4504f12d4447ba80f01cbb31b8c0a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.877221 5841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d26c0973a2dc4c639bc3a616e28fbfef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.878770 5973 tablet_service.cc:1467] Processing CreateTablet for tablet 7f4b2166aae54490a36ced8d37caf7b1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f3b4504f12d4447ba80f01cbb31b8c0a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.878983 6603 tablet_bootstrap.cc:492] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a: Bootstrap starting.
I20240501 07:59:08.878993 6604 tablet_bootstrap.cc:492] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723: Bootstrap starting.
I20240501 07:59:08.879040 5973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f4b2166aae54490a36ced8d37caf7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.879693 6603 tablet_bootstrap.cc:654] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.879850 6604 tablet_bootstrap.cc:654] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.880944 5972 tablet_service.cc:1467] Processing CreateTablet for tablet d26c0973a2dc4c639bc3a616e28fbfef (DEFAULT_TABLE table=rebalance_test_table_2 [id=f3b4504f12d4447ba80f01cbb31b8c0a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.881098 6605 tablet_bootstrap.cc:492] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33: Bootstrap starting.
I20240501 07:59:08.881170 5972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d26c0973a2dc4c639bc3a616e28fbfef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.881181 6603 tablet_bootstrap.cc:492] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a: No bootstrap required, opened a new log
I20240501 07:59:08.881183 6604 tablet_bootstrap.cc:492] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723: No bootstrap required, opened a new log
I20240501 07:59:08.881251 6603 ts_tablet_manager.cc:1367] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:08.881254 6604 ts_tablet_manager.cc:1367] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:08.881672 6604 raft_consensus.cc:348] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:08.881677 6603 raft_consensus.cc:348] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.881775 6604 raft_consensus.cc:374] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.881783 6603 raft_consensus.cc:374] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.881814 6603 raft_consensus.cc:729] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c782f15fb643acb2ef72105f1fb04a, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.881806 6604 raft_consensus.cc:729] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.881937 6603 consensus_queue.cc:260] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.881938 6604 consensus_queue.cc:260] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:08.882009 6605 tablet_bootstrap.cc:654] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.882190 6603 ts_tablet_manager.cc:1398] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.882195 6604 ts_tablet_manager.cc:1398] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:08.882359 6603 tablet_bootstrap.cc:492] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a: Bootstrap starting.
I20240501 07:59:08.883273 6605 tablet_bootstrap.cc:492] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33: No bootstrap required, opened a new log
I20240501 07:59:08.883368 6605 ts_tablet_manager.cc:1367] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:08.883716 6605 raft_consensus.cc:348] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.883812 6605 raft_consensus.cc:374] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.883839 6605 raft_consensus.cc:729] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.883949 6605 consensus_queue.cc:260] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:08.884183 6605 ts_tablet_manager.cc:1398] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:08.884330 6605 tablet_bootstrap.cc:492] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33: Bootstrap starting.
I20240501 07:59:08.885233 6605 tablet_bootstrap.cc:654] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.885684 6603 tablet_bootstrap.cc:654] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.886256 6604 tablet_bootstrap.cc:492] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723: Bootstrap starting.
I20240501 07:59:08.886902 6605 tablet_bootstrap.cc:492] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33: No bootstrap required, opened a new log
I20240501 07:59:08.886982 6605 ts_tablet_manager.cc:1367] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:08.887023 6603 tablet_bootstrap.cc:492] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a: No bootstrap required, opened a new log
I20240501 07:59:08.887101 6603 ts_tablet_manager.cc:1367] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:08.887166 6604 tablet_bootstrap.cc:654] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.887420 6605 raft_consensus.cc:348] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:08.887518 6605 raft_consensus.cc:374] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.887547 6605 raft_consensus.cc:729] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.887532 6603 raft_consensus.cc:348] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:08.887636 6603 raft_consensus.cc:374] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.887665 6603 raft_consensus.cc:729] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c782f15fb643acb2ef72105f1fb04a, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.887665 6605 consensus_queue.cc:260] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:08.887792 6603 consensus_queue.cc:260] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:08.887925 6605 ts_tablet_manager.cc:1398] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:08.888043 6603 ts_tablet_manager.cc:1398] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.888504 6604 tablet_bootstrap.cc:492] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723: No bootstrap required, opened a new log
I20240501 07:59:08.888587 6604 ts_tablet_manager.cc:1367] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:08.889011 6604 raft_consensus.cc:348] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:08.889113 6604 raft_consensus.cc:374] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.889153 6604 raft_consensus.cc:729] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.889256 6604 consensus_queue.cc:260] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:08.889508 6604 ts_tablet_manager.cc:1398] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.929119 6613 raft_consensus.cc:482] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:08.929250 6613 raft_consensus.cc:504] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:08.929594 6613 leader_election.cc:288] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797)
I20240501 07:59:08.930052 5731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "662d37043f054aa1a562cde8d6c155a9" candidate_uuid: "f022e3632e9a428882a6cd16dc398723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c782f15fb643acb2ef72105f1fb04a" is_pre_election: true
I20240501 07:59:08.930192 5731 raft_consensus.cc:2454] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f022e3632e9a428882a6cd16dc398723 in term 0.
I20240501 07:59:08.930527 5797 leader_election.cc:302] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:08.930696 6613 raft_consensus.cc:2781] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:08.930752 6613 raft_consensus.cc:482] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:08.930784 6613 raft_consensus.cc:3037] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.931586 6613 raft_consensus.cc:504] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:08.931869 6613 leader_election.cc:288] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 election: Requested vote from peers 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797)
I20240501 07:59:08.932374 5731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "662d37043f054aa1a562cde8d6c155a9" candidate_uuid: "f022e3632e9a428882a6cd16dc398723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c782f15fb643acb2ef72105f1fb04a"
I20240501 07:59:08.932507 5731 raft_consensus.cc:3037] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.933252 5731 raft_consensus.cc:2454] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f022e3632e9a428882a6cd16dc398723 in term 1.
I20240501 07:59:08.933529 5797 leader_election.cc:302] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:08.933722 6613 raft_consensus.cc:2781] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:08.933820 6613 raft_consensus.cc:686] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Becoming Leader. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Running, Role: LEADER
I20240501 07:59:08.933949 6613 consensus_queue.cc:237] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:08.935547 5583 catalog_manager.cc:5484] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 reported cstate change: term changed from 0 to 1, leader changed from <none> to f022e3632e9a428882a6cd16dc398723 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } }
I20240501 07:59:09.002559 6613 raft_consensus.cc:482] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:09.002717 6613 raft_consensus.cc:504] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:09.003084 6613 leader_election.cc:288] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531)
I20240501 07:59:09.003597 5993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26c0973a2dc4c639bc3a616e28fbfef" candidate_uuid: "f022e3632e9a428882a6cd16dc398723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec372f1e72b040e780e6bdd4d5409d33" is_pre_election: true
I20240501 07:59:09.003738 5993 raft_consensus.cc:2454] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f022e3632e9a428882a6cd16dc398723 in term 0.
I20240501 07:59:09.004040 5797 leader_election.cc:302] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ec372f1e72b040e780e6bdd4d5409d33, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:09.004244 6613 raft_consensus.cc:2781] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:09.004302 6613 raft_consensus.cc:482] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:09.004333 6613 raft_consensus.cc:3037] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.006570 6613 raft_consensus.cc:504] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:09.006961 6613 leader_election.cc:288] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 election: Requested vote from peers ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531)
I20240501 07:59:09.007638 5993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26c0973a2dc4c639bc3a616e28fbfef" candidate_uuid: "f022e3632e9a428882a6cd16dc398723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec372f1e72b040e780e6bdd4d5409d33"
I20240501 07:59:09.007771 5993 raft_consensus.cc:3037] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.008519 5993 raft_consensus.cc:2454] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f022e3632e9a428882a6cd16dc398723 in term 1.
I20240501 07:59:09.008941 5797 leader_election.cc:302] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ec372f1e72b040e780e6bdd4d5409d33, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:09.009145 6613 raft_consensus.cc:2781] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:09.009231 6613 raft_consensus.cc:686] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Becoming Leader. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Running, Role: LEADER
I20240501 07:59:09.009387 6613 consensus_queue.cc:237] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:09.011341 5588 catalog_manager.cc:5484] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 reported cstate change: term changed from 0 to 1, leader changed from <none> to f022e3632e9a428882a6cd16dc398723 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } }
I20240501 07:59:09.025839 6611 raft_consensus.cc:482] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:09.025985 6611 raft_consensus.cc:504] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.026369 6611 leader_election.cc:288] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797)
I20240501 07:59:09.026903 5731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b2166aae54490a36ced8d37caf7b1" candidate_uuid: "ec372f1e72b040e780e6bdd4d5409d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c782f15fb643acb2ef72105f1fb04a" is_pre_election: true
I20240501 07:59:09.027045 5731 raft_consensus.cc:2454] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec372f1e72b040e780e6bdd4d5409d33 in term 0.
I20240501 07:59:09.027416 5928 leader_election.cc:302] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, ec372f1e72b040e780e6bdd4d5409d33; no voters:
I20240501 07:59:09.027580 6611 raft_consensus.cc:2781] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:09.027639 6611 raft_consensus.cc:482] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:09.027670 6611 raft_consensus.cc:3037] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.028465 6611 raft_consensus.cc:504] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.028764 6611 leader_election.cc:288] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 election: Requested vote from peers 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797)
I20240501 07:59:09.029278 5731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b2166aae54490a36ced8d37caf7b1" candidate_uuid: "ec372f1e72b040e780e6bdd4d5409d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c782f15fb643acb2ef72105f1fb04a"
I20240501 07:59:09.029407 5731 raft_consensus.cc:3037] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.030198 5731 raft_consensus.cc:2454] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec372f1e72b040e780e6bdd4d5409d33 in term 1.
I20240501 07:59:09.030550 5928 leader_election.cc:302] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, ec372f1e72b040e780e6bdd4d5409d33; no voters:
I20240501 07:59:09.030717 6611 raft_consensus.cc:2781] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:09.030800 6611 raft_consensus.cc:686] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Becoming Leader. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Running, Role: LEADER
I20240501 07:59:09.030942 6611 consensus_queue.cc:237] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.032596 5588 catalog_manager.cc:5484] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } }
W20240501 07:59:09.152944 6722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:09.153249 6722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:09.156577 6722 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 07:59:09.249887 5731 raft_consensus.cc:1261] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:09.250329 6611 consensus_queue.cc:1035] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.254165 5730 raft_consensus.cc:1261] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:09.254930 6613 consensus_queue.cc:1035] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.255106 5993 raft_consensus.cc:1261] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:09.256347 6613 consensus_queue.cc:1035] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 97.8407 ms
time per row: 0.0489204 ms
I20240501 07:59:09.298624 5582 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40744:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:09.304826 5972 tablet_service.cc:1467] Processing CreateTablet for tablet 0776eb886275492285da625d69e77e6b (DEFAULT_TABLE table=rebalance_test_table_3 [id=62d90ece49d2427fbd192ff552b40579]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.304826 5842 tablet_service.cc:1467] Processing CreateTablet for tablet dad26af7f84e42208fd4461656117035 (DEFAULT_TABLE table=rebalance_test_table_3 [id=62d90ece49d2427fbd192ff552b40579]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.305038 5711 tablet_service.cc:1467] Processing CreateTablet for tablet dad26af7f84e42208fd4461656117035 (DEFAULT_TABLE table=rebalance_test_table_3 [id=62d90ece49d2427fbd192ff552b40579]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.305182 5972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0776eb886275492285da625d69e77e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.305182 5842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad26af7f84e42208fd4461656117035. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.305544 5711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad26af7f84e42208fd4461656117035. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.307387 6605 tablet_bootstrap.cc:492] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33: Bootstrap starting.
I20240501 07:59:09.307749 5710 tablet_service.cc:1467] Processing CreateTablet for tablet 0776eb886275492285da625d69e77e6b (DEFAULT_TABLE table=rebalance_test_table_3 [id=62d90ece49d2427fbd192ff552b40579]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.307902 5972 tablet_service.cc:1467] Processing CreateTablet for tablet a6cb1009e4fb4fb0a8e7d55c2c24ff65 (DEFAULT_TABLE table=rebalance_test_table_3 [id=62d90ece49d2427fbd192ff552b40579]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.308018 5710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0776eb886275492285da625d69e77e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.308089 5972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6cb1009e4fb4fb0a8e7d55c2c24ff65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.308470 6605 tablet_bootstrap.cc:654] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.309232 6604 tablet_bootstrap.cc:492] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723: Bootstrap starting.
I20240501 07:59:09.310160 6604 tablet_bootstrap.cc:654] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.310335 5841 tablet_service.cc:1467] Processing CreateTablet for tablet a6cb1009e4fb4fb0a8e7d55c2c24ff65 (DEFAULT_TABLE table=rebalance_test_table_3 [id=62d90ece49d2427fbd192ff552b40579]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.310662 5841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6cb1009e4fb4fb0a8e7d55c2c24ff65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.311411 6604 tablet_bootstrap.cc:492] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723: No bootstrap required, opened a new log
I20240501 07:59:09.311470 6605 tablet_bootstrap.cc:492] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33: No bootstrap required, opened a new log
I20240501 07:59:09.311494 6604 ts_tablet_manager.cc:1367] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:09.311569 6605 ts_tablet_manager.cc:1367] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:09.311887 6604 raft_consensus.cc:348] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.311983 6604 raft_consensus.cc:374] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.312012 6604 raft_consensus.cc:729] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.312122 6604 consensus_queue.cc:260] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.312142 6605 raft_consensus.cc:348] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.312280 6605 raft_consensus.cc:374] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.312311 6605 raft_consensus.cc:729] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.312398 6604 ts_tablet_manager.cc:1398] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:09.312429 6605 consensus_queue.cc:260] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.312719 6605 ts_tablet_manager.cc:1398] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.312907 6605 tablet_bootstrap.cc:492] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33: Bootstrap starting.
I20240501 07:59:09.312963 6603 tablet_bootstrap.cc:492] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a: Bootstrap starting.
I20240501 07:59:09.314080 6603 tablet_bootstrap.cc:654] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.314122 6605 tablet_bootstrap.cc:654] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.315095 6604 tablet_bootstrap.cc:492] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723: Bootstrap starting.
I20240501 07:59:09.315339 6603 tablet_bootstrap.cc:492] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a: No bootstrap required, opened a new log
I20240501 07:59:09.315440 6603 ts_tablet_manager.cc:1367] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:09.315990 6604 tablet_bootstrap.cc:654] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.315925 6603 raft_consensus.cc:348] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.316042 6603 raft_consensus.cc:374] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.316078 6603 raft_consensus.cc:729] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c782f15fb643acb2ef72105f1fb04a, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.316232 6603 consensus_queue.cc:260] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.316534 6603 ts_tablet_manager.cc:1398] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:09.316859 6603 tablet_bootstrap.cc:492] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a: Bootstrap starting.
I20240501 07:59:09.318212 6603 tablet_bootstrap.cc:654] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.318383 6604 tablet_bootstrap.cc:492] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723: No bootstrap required, opened a new log
I20240501 07:59:09.318550 6604 ts_tablet_manager.cc:1367] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 07:59:09.319270 6604 raft_consensus.cc:348] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:09.320065 6604 raft_consensus.cc:374] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.320116 6604 raft_consensus.cc:729] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.320242 6604 consensus_queue.cc:260] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:09.320444 6605 tablet_bootstrap.cc:492] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33: No bootstrap required, opened a new log
I20240501 07:59:09.320560 6605 ts_tablet_manager.cc:1367] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:59:09.320562 6604 ts_tablet_manager.cc:1398] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:09.321097 6605 raft_consensus.cc:348] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:09.321225 6605 raft_consensus.cc:374] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.321283 6605 raft_consensus.cc:729] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.321439 6605 consensus_queue.cc:260] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:09.321799 6605 ts_tablet_manager.cc:1398] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:09.322796 6603 tablet_bootstrap.cc:492] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a: No bootstrap required, opened a new log
I20240501 07:59:09.322906 6603 ts_tablet_manager.cc:1367] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 07:59:09.323397 6603 raft_consensus.cc:348] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.323516 6603 raft_consensus.cc:374] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.323556 6603 raft_consensus.cc:729] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c782f15fb643acb2ef72105f1fb04a, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.323688 6603 consensus_queue.cc:260] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.324005 6603 ts_tablet_manager.cc:1398] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 07:59:09.438223 6712 raft_consensus.cc:482] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:09.438355 6712 raft_consensus.cc:504] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.438735 6712 leader_election.cc:288] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187)
I20240501 07:59:09.439194 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad26af7f84e42208fd4461656117035" candidate_uuid: "46c782f15fb643acb2ef72105f1fb04a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f022e3632e9a428882a6cd16dc398723" is_pre_election: true
I20240501 07:59:09.439347 5862 raft_consensus.cc:2454] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c782f15fb643acb2ef72105f1fb04a in term 0.
I20240501 07:59:09.439656 5664 leader_election.cc:302] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:09.439826 6712 raft_consensus.cc:2781] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:09.439898 6712 raft_consensus.cc:482] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:09.439934 6712 raft_consensus.cc:3037] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.440919 6712 raft_consensus.cc:504] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.441233 6712 leader_election.cc:288] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [CANDIDATE]: Term 1 election: Requested vote from peers f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187)
I20240501 07:59:09.441654 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad26af7f84e42208fd4461656117035" candidate_uuid: "46c782f15fb643acb2ef72105f1fb04a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f022e3632e9a428882a6cd16dc398723"
I20240501 07:59:09.441769 5862 raft_consensus.cc:3037] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.442579 5862 raft_consensus.cc:2454] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c782f15fb643acb2ef72105f1fb04a in term 1.
I20240501 07:59:09.442878 5664 leader_election.cc:302] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:09.443029 6712 raft_consensus.cc:2781] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:09.443113 6712 raft_consensus.cc:686] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [term 1 LEADER]: Becoming Leader. State: Replica: 46c782f15fb643acb2ef72105f1fb04a, State: Running, Role: LEADER
I20240501 07:59:09.443249 6712 consensus_queue.cc:237] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.445024 5583 catalog_manager.cc:5484] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a reported cstate change: term changed from 0 to 1, leader changed from <none> to 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129). New cstate: current_term: 1 leader_uuid: "46c782f15fb643acb2ef72105f1fb04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } }
I20240501 07:59:09.450699 6611 raft_consensus.cc:482] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:09.450867 6611 raft_consensus.cc:504] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.451236 6611 leader_election.cc:288] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797)
I20240501 07:59:09.451777 5730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0776eb886275492285da625d69e77e6b" candidate_uuid: "ec372f1e72b040e780e6bdd4d5409d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c782f15fb643acb2ef72105f1fb04a" is_pre_election: true
I20240501 07:59:09.451907 5730 raft_consensus.cc:2454] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec372f1e72b040e780e6bdd4d5409d33 in term 0.
I20240501 07:59:09.452193 5928 leader_election.cc:302] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, ec372f1e72b040e780e6bdd4d5409d33; no voters:
I20240501 07:59:09.452384 6611 raft_consensus.cc:2781] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:09.452448 6611 raft_consensus.cc:482] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:09.452486 6611 raft_consensus.cc:3037] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.453316 6611 raft_consensus.cc:504] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.453603 6611 leader_election.cc:288] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 election: Requested vote from peers 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797)
I20240501 07:59:09.454020 5730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0776eb886275492285da625d69e77e6b" candidate_uuid: "ec372f1e72b040e780e6bdd4d5409d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c782f15fb643acb2ef72105f1fb04a"
I20240501 07:59:09.454129 5730 raft_consensus.cc:3037] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.454906 5730 raft_consensus.cc:2454] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec372f1e72b040e780e6bdd4d5409d33 in term 1.
I20240501 07:59:09.455197 5928 leader_election.cc:302] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, ec372f1e72b040e780e6bdd4d5409d33; no voters:
I20240501 07:59:09.455395 6611 raft_consensus.cc:2781] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:09.455474 6611 raft_consensus.cc:686] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Becoming Leader. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Running, Role: LEADER
I20240501 07:59:09.455610 6611 consensus_queue.cc:237] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.457139 5583 catalog_manager.cc:5484] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } }
I20240501 07:59:09.624655 6611 raft_consensus.cc:482] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:09.624825 6611 raft_consensus.cc:504] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:09.625244 6611 leader_election.cc:288] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187)
I20240501 07:59:09.625790 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6cb1009e4fb4fb0a8e7d55c2c24ff65" candidate_uuid: "ec372f1e72b040e780e6bdd4d5409d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f022e3632e9a428882a6cd16dc398723" is_pre_election: true
I20240501 07:59:09.625937 5862 raft_consensus.cc:2454] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec372f1e72b040e780e6bdd4d5409d33 in term 0.
I20240501 07:59:09.626237 5926 leader_election.cc:302] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ec372f1e72b040e780e6bdd4d5409d33, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:09.626479 6611 raft_consensus.cc:2781] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:09.626544 6611 raft_consensus.cc:482] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:09.626591 6611 raft_consensus.cc:3037] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.627462 6611 raft_consensus.cc:504] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:09.627801 6611 leader_election.cc:288] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 election: Requested vote from peers f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187)
I20240501 07:59:09.628285 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6cb1009e4fb4fb0a8e7d55c2c24ff65" candidate_uuid: "ec372f1e72b040e780e6bdd4d5409d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f022e3632e9a428882a6cd16dc398723"
I20240501 07:59:09.628405 5862 raft_consensus.cc:3037] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.629177 5862 raft_consensus.cc:2454] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec372f1e72b040e780e6bdd4d5409d33 in term 1.
I20240501 07:59:09.629491 5926 leader_election.cc:302] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ec372f1e72b040e780e6bdd4d5409d33, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:09.629675 6611 raft_consensus.cc:2781] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:09.629777 6611 raft_consensus.cc:686] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Becoming Leader. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Running, Role: LEADER
I20240501 07:59:09.629912 6611 consensus_queue.cc:237] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:09.631626 5583 catalog_manager.cc:5484] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } }
W20240501 07:59:09.790078 6762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:09.790362 6762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:09.793982 6762 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 07:59:09.886183 5862 raft_consensus.cc:1261] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Refusing update from remote peer 46c782f15fb643acb2ef72105f1fb04a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:09.886904 6712 consensus_queue.cc:1035] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.887459 5730 raft_consensus.cc:1261] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:09.887887 6611 consensus_queue.cc:1035] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.893872 5861 raft_consensus.cc:1261] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:09.894328 6616 consensus_queue.cc:1035] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 92.7303 ms
time per row: 0.0463652 ms
I20240501 07:59:09.936684 5582 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40744:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:09.943176 5710 tablet_service.cc:1467] Processing CreateTablet for tablet 877dfd1e30ee4e12b16674d96d57c48e (DEFAULT_TABLE table=rebalance_test_table_4 [id=2c9dba3a010e4d3fa551b89730285ff8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.943158 5841 tablet_service.cc:1467] Processing CreateTablet for tablet 877dfd1e30ee4e12b16674d96d57c48e (DEFAULT_TABLE table=rebalance_test_table_4 [id=2c9dba3a010e4d3fa551b89730285ff8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.943241 5972 tablet_service.cc:1467] Processing CreateTablet for tablet 2a577537e9294fd5bd8507232d1d1a76 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2c9dba3a010e4d3fa551b89730285ff8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.943492 5841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 877dfd1e30ee4e12b16674d96d57c48e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.943490 5710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 877dfd1e30ee4e12b16674d96d57c48e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.943444 5711 tablet_service.cc:1467] Processing CreateTablet for tablet 2a577537e9294fd5bd8507232d1d1a76 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2c9dba3a010e4d3fa551b89730285ff8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.943533 5972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a577537e9294fd5bd8507232d1d1a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.943667 5711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a577537e9294fd5bd8507232d1d1a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.945838 6814 tablet_bootstrap.cc:492] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723: Bootstrap starting.
I20240501 07:59:09.946281 5972 tablet_service.cc:1467] Processing CreateTablet for tablet 26ad4c1ac47544f586d3f46f5dc9c77e (DEFAULT_TABLE table=rebalance_test_table_4 [id=2c9dba3a010e4d3fa551b89730285ff8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.946599 5972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ad4c1ac47544f586d3f46f5dc9c77e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.946586 5841 tablet_service.cc:1467] Processing CreateTablet for tablet 26ad4c1ac47544f586d3f46f5dc9c77e (DEFAULT_TABLE table=rebalance_test_table_4 [id=2c9dba3a010e4d3fa551b89730285ff8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.946815 5841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ad4c1ac47544f586d3f46f5dc9c77e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.946956 6814 tablet_bootstrap.cc:654] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.947914 6815 tablet_bootstrap.cc:492] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a: Bootstrap starting.
I20240501 07:59:09.948271 6814 tablet_bootstrap.cc:492] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723: No bootstrap required, opened a new log
I20240501 07:59:09.948376 6814 ts_tablet_manager.cc:1367] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:09.948683 6816 tablet_bootstrap.cc:492] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33: Bootstrap starting.
I20240501 07:59:09.948859 6814 raft_consensus.cc:348] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:09.948990 6814 raft_consensus.cc:374] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.949040 6814 raft_consensus.cc:729] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.949178 6814 consensus_queue.cc:260] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:09.949275 6815 tablet_bootstrap.cc:654] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.949498 6816 tablet_bootstrap.cc:654] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.949497 6814 ts_tablet_manager.cc:1398] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.949674 6814 tablet_bootstrap.cc:492] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723: Bootstrap starting.
I20240501 07:59:09.950524 6814 tablet_bootstrap.cc:654] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.950641 6816 tablet_bootstrap.cc:492] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33: No bootstrap required, opened a new log
I20240501 07:59:09.950729 6816 ts_tablet_manager.cc:1367] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:09.951195 6816 raft_consensus.cc:348] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.951314 6816 raft_consensus.cc:374] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.951347 6816 raft_consensus.cc:729] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.951468 6816 consensus_queue.cc:260] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.951508 6814 tablet_bootstrap.cc:492] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723: No bootstrap required, opened a new log
I20240501 07:59:09.951575 6814 ts_tablet_manager.cc:1367] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:09.951669 6815 tablet_bootstrap.cc:492] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a: No bootstrap required, opened a new log
I20240501 07:59:09.951952 6815 ts_tablet_manager.cc:1367] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:09.951994 6816 ts_tablet_manager.cc:1398] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.951995 6814 raft_consensus.cc:348] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:09.952088 6814 raft_consensus.cc:374] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.952121 6814 raft_consensus.cc:729] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.952188 6816 tablet_bootstrap.cc:492] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33: Bootstrap starting.
I20240501 07:59:09.952235 6814 consensus_queue.cc:260] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:09.952450 6815 raft_consensus.cc:348] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.952544 6814 ts_tablet_manager.cc:1398] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:09.952558 6815 raft_consensus.cc:374] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.952587 6815 raft_consensus.cc:729] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c782f15fb643acb2ef72105f1fb04a, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.952710 6815 consensus_queue.cc:260] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:09.952967 6815 ts_tablet_manager.cc:1398] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.953101 6816 tablet_bootstrap.cc:654] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.953148 6815 tablet_bootstrap.cc:492] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a: Bootstrap starting.
I20240501 07:59:09.954278 6815 tablet_bootstrap.cc:654] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.956203 6816 tablet_bootstrap.cc:492] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33: No bootstrap required, opened a new log
I20240501 07:59:09.956291 6816 ts_tablet_manager.cc:1367] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:09.956753 6816 raft_consensus.cc:348] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:09.956859 6816 raft_consensus.cc:374] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.956881 6816 raft_consensus.cc:729] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.956914 6815 tablet_bootstrap.cc:492] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a: No bootstrap required, opened a new log
I20240501 07:59:09.957003 6815 ts_tablet_manager.cc:1367] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:09.956988 6816 consensus_queue.cc:260] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:09.957247 6816 ts_tablet_manager.cc:1398] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.957417 6815 raft_consensus.cc:348] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:09.957501 6815 raft_consensus.cc:374] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.957530 6815 raft_consensus.cc:729] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c782f15fb643acb2ef72105f1fb04a, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.957648 6815 consensus_queue.cc:260] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:09.957875 6815 ts_tablet_manager.cc:1398] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:10.008289 6620 raft_consensus.cc:482] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:10.008433 6620 raft_consensus.cc:504] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:10.008791 6620 leader_election.cc:288] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797)
I20240501 07:59:10.009335 5730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "877dfd1e30ee4e12b16674d96d57c48e" candidate_uuid: "f022e3632e9a428882a6cd16dc398723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c782f15fb643acb2ef72105f1fb04a" is_pre_election: true
I20240501 07:59:10.009471 5730 raft_consensus.cc:2454] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f022e3632e9a428882a6cd16dc398723 in term 0.
I20240501 07:59:10.009819 5797 leader_election.cc:302] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:10.010016 6620 raft_consensus.cc:2781] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:10.010071 6620 raft_consensus.cc:482] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:10.010099 6620 raft_consensus.cc:3037] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.010960 6620 raft_consensus.cc:504] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:10.011350 6620 leader_election.cc:288] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 election: Requested vote from peers 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797)
I20240501 07:59:10.011842 5730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "877dfd1e30ee4e12b16674d96d57c48e" candidate_uuid: "f022e3632e9a428882a6cd16dc398723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c782f15fb643acb2ef72105f1fb04a"
I20240501 07:59:10.011967 5730 raft_consensus.cc:3037] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.012895 5730 raft_consensus.cc:2454] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f022e3632e9a428882a6cd16dc398723 in term 1.
I20240501 07:59:10.013226 5797 leader_election.cc:302] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:10.013417 6620 raft_consensus.cc:2781] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:10.013630 6620 raft_consensus.cc:686] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Becoming Leader. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Running, Role: LEADER
I20240501 07:59:10.013782 6620 consensus_queue.cc:237] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } }
I20240501 07:59:10.015470 5586 catalog_manager.cc:5484] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 reported cstate change: term changed from 0 to 1, leader changed from <none> to f022e3632e9a428882a6cd16dc398723 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } }
I20240501 07:59:10.122717 6620 raft_consensus.cc:482] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:10.122871 6620 raft_consensus.cc:504] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:10.123209 6620 leader_election.cc:288] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531)
I20240501 07:59:10.123708 5993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ad4c1ac47544f586d3f46f5dc9c77e" candidate_uuid: "f022e3632e9a428882a6cd16dc398723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec372f1e72b040e780e6bdd4d5409d33" is_pre_election: true
I20240501 07:59:10.123868 5993 raft_consensus.cc:2454] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f022e3632e9a428882a6cd16dc398723 in term 0.
I20240501 07:59:10.124197 5797 leader_election.cc:302] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ec372f1e72b040e780e6bdd4d5409d33, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:10.124420 6620 raft_consensus.cc:2781] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:10.124480 6620 raft_consensus.cc:482] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:10.124507 6620 raft_consensus.cc:3037] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.125403 6620 raft_consensus.cc:504] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:10.125705 6620 leader_election.cc:288] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 election: Requested vote from peers ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531)
I20240501 07:59:10.126142 5993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ad4c1ac47544f586d3f46f5dc9c77e" candidate_uuid: "f022e3632e9a428882a6cd16dc398723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec372f1e72b040e780e6bdd4d5409d33"
I20240501 07:59:10.126258 5993 raft_consensus.cc:3037] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.127066 5993 raft_consensus.cc:2454] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f022e3632e9a428882a6cd16dc398723 in term 1.
I20240501 07:59:10.127377 5797 leader_election.cc:302] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ec372f1e72b040e780e6bdd4d5409d33, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:10.127564 6620 raft_consensus.cc:2781] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:10.127653 6620 raft_consensus.cc:686] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Becoming Leader. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Running, Role: LEADER
I20240501 07:59:10.127786 6620 consensus_queue.cc:237] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } }
I20240501 07:59:10.129557 5586 catalog_manager.cc:5484] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 reported cstate change: term changed from 0 to 1, leader changed from <none> to f022e3632e9a428882a6cd16dc398723 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } }
I20240501 07:59:10.152995 6611 raft_consensus.cc:482] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:10.153138 6611 raft_consensus.cc:504] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:10.153496 6611 leader_election.cc:288] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797)
I20240501 07:59:10.154086 5730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a577537e9294fd5bd8507232d1d1a76" candidate_uuid: "ec372f1e72b040e780e6bdd4d5409d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c782f15fb643acb2ef72105f1fb04a" is_pre_election: true
I20240501 07:59:10.154223 5730 raft_consensus.cc:2454] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec372f1e72b040e780e6bdd4d5409d33 in term 0.
I20240501 07:59:10.154551 5928 leader_election.cc:302] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, ec372f1e72b040e780e6bdd4d5409d33; no voters:
I20240501 07:59:10.154732 6611 raft_consensus.cc:2781] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:10.154788 6611 raft_consensus.cc:482] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:10.154825 6611 raft_consensus.cc:3037] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.155570 6611 raft_consensus.cc:504] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:10.155859 6611 leader_election.cc:288] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 election: Requested vote from peers 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797)
I20240501 07:59:10.156342 5730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a577537e9294fd5bd8507232d1d1a76" candidate_uuid: "ec372f1e72b040e780e6bdd4d5409d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c782f15fb643acb2ef72105f1fb04a"
I20240501 07:59:10.156474 5730 raft_consensus.cc:3037] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.157265 5730 raft_consensus.cc:2454] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec372f1e72b040e780e6bdd4d5409d33 in term 1.
I20240501 07:59:10.157615 5928 leader_election.cc:302] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, ec372f1e72b040e780e6bdd4d5409d33; no voters:
I20240501 07:59:10.157791 6611 raft_consensus.cc:2781] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:10.157891 6611 raft_consensus.cc:686] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Becoming Leader. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Running, Role: LEADER
I20240501 07:59:10.158031 6611 consensus_queue.cc:237] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } }
I20240501 07:59:10.159706 5586 catalog_manager.cc:5484] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } }
W20240501 07:59:10.294941 6827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:10.295222 6827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:10.298497 6827 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 07:59:10.378759 5730 raft_consensus.cc:1261] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:10.379264 6611 consensus_queue.cc:1035] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:10.384611 5993 raft_consensus.cc:1261] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:10.384630 5730 raft_consensus.cc:1261] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:10.385069 6620 consensus_queue.cc:1035] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:10.390326 6620 consensus_queue.cc:1035] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 84.2523 ms
time per row: 0.0421262 ms
I20240501 07:59:10.428567 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:38041
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:40255
--webserver_port=33881
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 07:59:10.543740 6858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:10.544003 6858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:10.544047 6858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:10.547312 6858 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:10.547399 6858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:10.547477 6858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:59:10.551339 6858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:38041
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=33881
--tserver_master_addrs=127.0.99.190:40255
--never_fsync=true
--heap_profile_path=/tmp/kudu.6858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:10.552332 6858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:10.553510 6858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:10.562459 6863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:10.562836 6864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:10.563406 6866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:10.564008 6858 server_base.cc:1034] running on GCE node
I20240501 07:59:10.564486 6858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:10.565167 6858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:10.566346 6858 hybrid_clock.cc:648] HybridClock initialized: now 1714550350566323 us; error 42 us; skew 500 ppm
I20240501 07:59:10.569096 6858 webserver.cc:460] Webserver started at http://127.0.99.132:33881/ using document root <none> and password file <none>
I20240501 07:59:10.569793 6858 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:10.569883 6858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:10.573740 6858 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:10.576131 6872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:10.577418 6858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:10.577546 6858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "12f3f54871994efeb833126452c61da9"
format_stamp: "Formatted at 2024-05-01 07:59:02 on dist-test-slave-2srn"
I20240501 07:59:10.577992 6858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:10.605373 6858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:10.606076 6858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:10.606254 6858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:10.606884 6858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:10.607842 6858 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:10.607893 6858 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:10.607949 6858 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:10.607976 6858 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:10.617108 6858 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:38041
I20240501 07:59:10.617136 6985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:38041 every 8 connection(s)
I20240501 07:59:10.618225 6858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:59:10.625474 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 6858
I20240501 07:59:10.625890 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:34267
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:40255
--webserver_port=44059
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:10.628211 6986 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40255
I20240501 07:59:10.628549 6986 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:10.629366 6986 heartbeater.cc:511] Master 127.0.99.190:40255 requested a full tablet report, sending...
I20240501 07:59:10.630971 5583 ts_manager.cc:194] Re-registered known tserver with Master: 12f3f54871994efeb833126452c61da9 (127.0.99.132:38041)
I20240501 07:59:10.631809 5583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:39073
W20240501 07:59:10.737035 6989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:10.737289 6989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:10.737339 6989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:10.740514 6989 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:10.740594 6989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:10.740664 6989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:59:10.745348 6989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:34267
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=44059
--tserver_master_addrs=127.0.99.190:40255
--never_fsync=true
--heap_profile_path=/tmp/kudu.6989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:10.746312 6989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:10.747713 6989 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:10.756186 6994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:10.756655 6995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:10.757107 6997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:10.757654 6989 server_base.cc:1034] running on GCE node
I20240501 07:59:10.758069 6989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:10.758684 6989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:10.759853 6989 hybrid_clock.cc:648] HybridClock initialized: now 1714550350759828 us; error 40 us; skew 500 ppm
I20240501 07:59:10.761890 6989 webserver.cc:460] Webserver started at http://127.0.99.133:44059/ using document root <none> and password file <none>
I20240501 07:59:10.762492 6989 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:10.762567 6989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:10.766070 6989 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:10.768298 7003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:10.769371 6989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:10.769481 6989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "72a601327d9d447aa7957cde6c36e000"
format_stamp: "Formatted at 2024-05-01 07:59:03 on dist-test-slave-2srn"
I20240501 07:59:10.769928 6989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:10.786876 6989 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:10.787632 6989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:10.787794 6989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:10.788378 6989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:10.789341 6989 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:10.789391 6989 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:10.789433 6989 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:10.789464 6989 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:10.798703 6989 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:34267
I20240501 07:59:10.798748 7116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:34267 every 8 connection(s)
I20240501 07:59:10.799758 6989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:59:10.802233 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 6989
I20240501 07:59:10.802628 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:38017
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:40255
--webserver_port=36825
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:10.811198 7117 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40255
I20240501 07:59:10.811477 7117 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:10.812052 7117 heartbeater.cc:511] Master 127.0.99.190:40255 requested a full tablet report, sending...
I20240501 07:59:10.813242 5583 ts_manager.cc:194] Re-registered known tserver with Master: 72a601327d9d447aa7957cde6c36e000 (127.0.99.133:34267)
I20240501 07:59:10.813848 5583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:54093
W20240501 07:59:10.929636 7120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:10.929886 7120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:10.929950 7120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:10.933231 7120 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:10.933328 7120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:10.933426 7120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 07:59:10.937400 7120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:38017
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=36825
--tserver_master_addrs=127.0.99.190:40255
--never_fsync=true
--heap_profile_path=/tmp/kudu.7120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:10.938410 7120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:10.939630 7120 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:10.948522 7125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:10.949041 7128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:10.949414 7126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:10.950064 7120 server_base.cc:1034] running on GCE node
I20240501 07:59:10.950568 7120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:10.951203 7120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:10.952358 7120 hybrid_clock.cc:648] HybridClock initialized: now 1714550350952346 us; error 33 us; skew 500 ppm
I20240501 07:59:10.954505 7120 webserver.cc:460] Webserver started at http://127.0.99.134:36825/ using document root <none> and password file <none>
I20240501 07:59:10.955096 7120 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:10.955169 7120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:10.958745 7120 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20240501 07:59:10.960960 7134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:10.962061 7120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:10.962183 7120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "afc8c07c29354a04926027f57b8c6fab"
format_stamp: "Formatted at 2024-05-01 07:59:03 on dist-test-slave-2srn"
I20240501 07:59:10.962671 7120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:10.978539 7120 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:10.979276 7120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:10.979435 7120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:10.980036 7120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:10.981348 7141 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 07:59:10.984573 7120 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 07:59:10.984660 7120 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20240501 07:59:10.984731 7120 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 07:59:10.986296 7120 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 07:59:10.986361 7120 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240501 07:59:10.986681 7141 tablet_bootstrap.cc:492] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab: Bootstrap starting.
I20240501 07:59:10.996974 7120 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:38017
I20240501 07:59:10.996999 7248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:38017 every 8 connection(s)
I20240501 07:59:10.998113 7120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 07:59:11.000840 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 7120
I20240501 07:59:11.001302 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:45001
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:40255
--webserver_port=33295
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:11.003556 7141 log.cc:824] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:11.005195 7141 tablet_bootstrap.cc:492] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:11.005569 7141 tablet_bootstrap.cc:492] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab: Bootstrap complete.
I20240501 07:59:11.006752 7141 ts_tablet_manager.cc:1367] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.006s
I20240501 07:59:11.008080 7249 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40255
I20240501 07:59:11.008322 7249 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:11.009017 7249 heartbeater.cc:511] Master 127.0.99.190:40255 requested a full tablet report, sending...
I20240501 07:59:11.010134 7141 raft_consensus.cc:348] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:11.010545 7141 raft_consensus.cc:729] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc8c07c29354a04926027f57b8c6fab, State: Initialized, Role: FOLLOWER
I20240501 07:59:11.011176 5583 ts_manager.cc:194] Re-registered known tserver with Master: afc8c07c29354a04926027f57b8c6fab (127.0.99.134:38017)
I20240501 07:59:11.011164 7141 consensus_queue.cc:260] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:11.012135 7141 ts_tablet_manager.cc:1398] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20240501 07:59:11.012144 5583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:48375
I20240501 07:59:11.014024 7249 heartbeater.cc:503] Master 127.0.99.190:40255 was elected leader, sending a full tablet report...
W20240501 07:59:11.112676 7252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:11.112933 7252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:11.112978 7252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:11.116142 7252 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:11.116232 7252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:11.116307 7252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 07:59:11.120174 7252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:45001
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=33295
--tserver_master_addrs=127.0.99.190:40255
--never_fsync=true
--heap_profile_path=/tmp/kudu.7252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:11.121167 7252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:11.122335 7252 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:11.131153 7260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:11.131655 7262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:11.132089 7259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:11.132710 7252 server_base.cc:1034] running on GCE node
I20240501 07:59:11.133237 7252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:11.133932 7252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:11.135108 7252 hybrid_clock.cc:648] HybridClock initialized: now 1714550351135091 us; error 29 us; skew 500 ppm
I20240501 07:59:11.137163 7252 webserver.cc:460] Webserver started at http://127.0.99.135:33295/ using document root <none> and password file <none>
I20240501 07:59:11.137739 7252 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:11.137816 7252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:11.141340 7252 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:11.143538 7268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:11.145320 7252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:59:11.145526 7252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a3af86b0e9a04ecbb15d305c48d7acb4"
format_stamp: "Formatted at 2024-05-01 07:59:03 on dist-test-slave-2srn"
I20240501 07:59:11.146158 7252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:11.163765 7252 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:11.164448 7252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:11.164608 7252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:11.165202 7252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:11.166450 7275 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 07:59:11.169821 7252 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 07:59:11.169906 7252 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20240501 07:59:11.169981 7252 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 07:59:11.171636 7252 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 07:59:11.171696 7252 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:11.171989 7275 tablet_bootstrap.cc:492] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4: Bootstrap starting.
I20240501 07:59:11.182600 7252 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:45001
I20240501 07:59:11.182687 7382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:45001 every 8 connection(s)
I20240501 07:59:11.183771 7252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 07:59:11.187927 7275 log.cc:824] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:11.188094 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 7252
I20240501 07:59:11.188282 398 test_util.cc:268] Using random seed: -593374920
I20240501 07:59:11.190876 7275 tablet_bootstrap.cc:492] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:59:11.191299 7275 tablet_bootstrap.cc:492] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4: Bootstrap complete.
I20240501 07:59:11.192378 7275 ts_tablet_manager.cc:1367] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.008s
I20240501 07:59:11.196019 7275 raft_consensus.cc:348] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:11.196740 7275 raft_consensus.cc:729] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3af86b0e9a04ecbb15d305c48d7acb4, State: Initialized, Role: FOLLOWER
I20240501 07:59:11.197301 7275 consensus_queue.cc:260] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:11.198226 7275 ts_tablet_manager.cc:1398] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 07:59:11.198465 7383 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40255
I20240501 07:59:11.198688 7383 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:11.199295 7383 heartbeater.cc:511] Master 127.0.99.190:40255 requested a full tablet report, sending...
I20240501 07:59:11.201252 5583 ts_manager.cc:194] Re-registered known tserver with Master: a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135:45001)
I20240501 07:59:11.201876 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240501 07:59:11.202105 398 test_util.cc:268] Using random seed: -593361097
I20240501 07:59:11.202068 5583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:54941
I20240501 07:59:11.204419 7383 heartbeater.cc:503] Master 127.0.99.190:40255 was elected leader, sending a full tablet report...
I20240501 07:59:11.213371 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240501 07:59:11.213655 398 test_util.cc:268] Using random seed: -593349547
I20240501 07:59:11.225395 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240501 07:59:11.225581 398 test_util.cc:268] Using random seed: -593337621
I20240501 07:59:11.237206 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240501 07:59:11.237396 398 test_util.cc:268] Using random seed: -593325807
W20240501 07:59:11.249382 7250 tablet.cc:2369] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:11.259804 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20240501 07:59:11.434703 7384 tablet.cc:2369] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:11.634598 6986 heartbeater.cc:503] Master 127.0.99.190:40255 was elected leader, sending a full tablet report...
I20240501 07:59:11.742439 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:11.750788 7051 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:11.753409 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:11.768785 7317 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:11.771342 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:11.778139 5972 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:11.785725 5841 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:11.816702 7117 heartbeater.cc:503] Master 127.0.99.190:40255 was elected leader, sending a full tablet report...
I20240501 07:59:12.101846 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:12.110184 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:12.120173 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:12.133006 7051 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:12.146340 5841 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:12.146852 7317 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:12.153024 5972 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:12.290602 7568 raft_consensus.cc:482] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:12.290825 7568 raft_consensus.cc:504] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:12.293342 7568 leader_election.cc:288] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135:45001)
I20240501 07:59:12.301898 7336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba935cdc94cf4eeab62cfee9e789e2eb" candidate_uuid: "afc8c07c29354a04926027f57b8c6fab" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" is_pre_election: true
I20240501 07:59:12.302274 7336 raft_consensus.cc:2396] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate afc8c07c29354a04926027f57b8c6fab for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240501 07:59:12.303025 7135 leader_election.cc:302] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: afc8c07c29354a04926027f57b8c6fab; no voters: a3af86b0e9a04ecbb15d305c48d7acb4
I20240501 07:59:12.304754 7568 raft_consensus.cc:2726] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:12.383293 7581 raft_consensus.cc:482] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:12.383502 7581 raft_consensus.cc:504] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:12.385681 7581 leader_election.cc:288] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers afc8c07c29354a04926027f57b8c6fab (127.0.99.134:38017)
I20240501 07:59:12.395633 5993 consensus_queue.cc:237] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } }
I20240501 07:59:12.397199 5728 raft_consensus.cc:1261] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 40. (index mismatch)
I20240501 07:59:12.398896 6611 consensus_queue.cc:1035] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20240501 07:59:12.401688 6611 raft_consensus.cc:2932] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Committing config change with OpId 1.40: config changed from index -1 to 40, VOTER 12f3f54871994efeb833126452c61da9 (127.0.99.132) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:12.403141 5730 raft_consensus.cc:2932] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Committing config change with OpId 1.40: config changed from index -1 to 40, VOTER 12f3f54871994efeb833126452c61da9 (127.0.99.132) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:12.406075 5588 catalog_manager.cc:5484] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a reported cstate change: config changed from index -1 to 40, VOTER 12f3f54871994efeb833126452c61da9 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
W20240501 07:59:12.422398 5929 consensus_peers.cc:485] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 -> Peer 12f3f54871994efeb833126452c61da9 (127.0.99.132:38041): Couldn't send request to peer 12f3f54871994efeb833126452c61da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f4b2166aae54490a36ced8d37caf7b1. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:12.428941 7203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba935cdc94cf4eeab62cfee9e789e2eb" candidate_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "afc8c07c29354a04926027f57b8c6fab" is_pre_election: true
I20240501 07:59:12.429175 7203 raft_consensus.cc:2454] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3af86b0e9a04ecbb15d305c48d7acb4 in term 1.
I20240501 07:59:12.429632 7269 leader_election.cc:302] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a3af86b0e9a04ecbb15d305c48d7acb4, afc8c07c29354a04926027f57b8c6fab; no voters:
I20240501 07:59:12.430089 7581 raft_consensus.cc:2781] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:12.430162 7581 raft_consensus.cc:482] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:12.430199 7581 raft_consensus.cc:3037] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:12.432549 7581 raft_consensus.cc:504] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:12.433378 7581 leader_election.cc:288] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [CANDIDATE]: Term 2 election: Requested vote from peers afc8c07c29354a04926027f57b8c6fab (127.0.99.134:38017)
I20240501 07:59:12.433372 7203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba935cdc94cf4eeab62cfee9e789e2eb" candidate_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "afc8c07c29354a04926027f57b8c6fab"
I20240501 07:59:12.433517 7203 raft_consensus.cc:3037] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:12.435796 7203 raft_consensus.cc:2454] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3af86b0e9a04ecbb15d305c48d7acb4 in term 2.
I20240501 07:59:12.436725 7269 leader_election.cc:302] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a3af86b0e9a04ecbb15d305c48d7acb4, afc8c07c29354a04926027f57b8c6fab; no voters:
I20240501 07:59:12.437163 7581 raft_consensus.cc:2781] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:12.437394 7581 raft_consensus.cc:686] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [term 2 LEADER]: Becoming Leader. State: Replica: a3af86b0e9a04ecbb15d305c48d7acb4, State: Running, Role: LEADER
I20240501 07:59:12.437772 7581 consensus_queue.cc:237] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:12.441867 5586 catalog_manager.cc:5484] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:12.456924 5728 consensus_queue.cc:237] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 1.49, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:12.458230 5861 raft_consensus.cc:1261] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Refusing update from remote peer 46c782f15fb643acb2ef72105f1fb04a: Log matching property violated. Preceding OpId in replica: term: 1 index: 49. Preceding OpId from leader: term: 1 index: 50. (index mismatch)
I20240501 07:59:12.459060 7469 consensus_queue.cc:1035] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20240501 07:59:12.460686 7469 raft_consensus.cc:2932] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [term 1 LEADER]: Committing config change with OpId 1.50: config changed from index -1 to 50, VOTER 72a601327d9d447aa7957cde6c36e000 (127.0.99.133) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:12.462643 5861 raft_consensus.cc:2932] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Committing config change with OpId 1.50: config changed from index -1 to 50, VOTER 72a601327d9d447aa7957cde6c36e000 (127.0.99.133) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:12.465770 5586 catalog_manager.cc:5484] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723 reported cstate change: config changed from index -1 to 50, VOTER 72a601327d9d447aa7957cde6c36e000 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "46c782f15fb643acb2ef72105f1fb04a" committed_config { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
W20240501 07:59:12.499253 5665 consensus_peers.cc:485] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a -> Peer 72a601327d9d447aa7957cde6c36e000 (127.0.99.133:34267): Couldn't send request to peer 72a601327d9d447aa7957cde6c36e000. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd95a7eff75845c0b49790c9657ef1e0. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:12.712934 5862 consensus_queue.cc:237] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } }
I20240501 07:59:12.719017 5992 raft_consensus.cc:1261] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 1 index: 41. (index mismatch)
I20240501 07:59:12.720696 7411 consensus_queue.cc:1035] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20240501 07:59:12.726226 6857 raft_consensus.cc:2932] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Committing config change with OpId 1.41: config changed from index -1 to 41, VOTER a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } }
I20240501 07:59:12.726750 5992 raft_consensus.cc:2932] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index -1 to 41, VOTER a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } }
I20240501 07:59:12.731106 5588 catalog_manager.cc:5484] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33 reported cstate change: config changed from index -1 to 41, VOTER a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } }
W20240501 07:59:12.756916 5795 consensus_peers.cc:485] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 -> Peer a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135:45001): Couldn't send request to peer a3af86b0e9a04ecbb15d305c48d7acb4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a07321bfc4c1445a906ec7c1261693e9. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:12.837155 5993 consensus_queue.cc:237] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 1.54, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:12.838707 5728 raft_consensus.cc:1261] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240501 07:59:12.840204 7422 consensus_queue.cc:1035] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20240501 07:59:12.842247 6611 raft_consensus.cc:2932] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Committing config change with OpId 1.55: config changed from index -1 to 55, VOTER afc8c07c29354a04926027f57b8c6fab (127.0.99.134) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
W20240501 07:59:12.843533 5926 consensus_peers.cc:485] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 -> Peer afc8c07c29354a04926027f57b8c6fab (127.0.99.134:38017): Couldn't send request to peer afc8c07c29354a04926027f57b8c6fab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f187bcb4c494805a8d6f366732b695a. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:12.847097 5728 raft_consensus.cc:2932] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index -1 to 55, VOTER afc8c07c29354a04926027f57b8c6fab (127.0.99.134) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:12.848297 5586 catalog_manager.cc:5484] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 reported cstate change: config changed from index -1 to 55, VOTER afc8c07c29354a04926027f57b8c6fab (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:12.868324 7639 ts_tablet_manager.cc:902] T bd95a7eff75845c0b49790c9657ef1e0 P 72a601327d9d447aa7957cde6c36e000: Initiating tablet copy from peer 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797)
I20240501 07:59:12.869092 7639 tablet_copy_client.cc:308] T bd95a7eff75845c0b49790c9657ef1e0 P 72a601327d9d447aa7957cde6c36e000: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33797
I20240501 07:59:12.881121 7203 raft_consensus.cc:1261] T ba935cdc94cf4eeab62cfee9e789e2eb P afc8c07c29354a04926027f57b8c6fab [term 2 FOLLOWER]: Refusing update from remote peer a3af86b0e9a04ecbb15d305c48d7acb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 07:59:12.884562 5751 tablet_copy_service.cc:140] P 46c782f15fb643acb2ef72105f1fb04a: Received BeginTabletCopySession request for tablet bd95a7eff75845c0b49790c9657ef1e0 from peer 72a601327d9d447aa7957cde6c36e000 ({username='slave'} at 127.0.99.133:49867)
I20240501 07:59:12.884718 5751 tablet_copy_service.cc:161] P 46c782f15fb643acb2ef72105f1fb04a: Beginning new tablet copy session on tablet bd95a7eff75845c0b49790c9657ef1e0 from peer 72a601327d9d447aa7957cde6c36e000 at {username='slave'} at 127.0.99.133:49867: session id = 72a601327d9d447aa7957cde6c36e000-bd95a7eff75845c0b49790c9657ef1e0
I20240501 07:59:12.886736 5751 tablet_copy_source_session.cc:205] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:12.888345 7581 consensus_queue.cc:1035] T ba935cdc94cf4eeab62cfee9e789e2eb P a3af86b0e9a04ecbb15d305c48d7acb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:12.892614 7639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd95a7eff75845c0b49790c9657ef1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.901072 7639 tablet_copy_client.cc:783] T bd95a7eff75845c0b49790c9657ef1e0 P 72a601327d9d447aa7957cde6c36e000: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:12.901970 7639 tablet_copy_client.cc:647] T bd95a7eff75845c0b49790c9657ef1e0 P 72a601327d9d447aa7957cde6c36e000: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:12.905264 7639 tablet_copy_client.cc:523] T bd95a7eff75845c0b49790c9657ef1e0 P 72a601327d9d447aa7957cde6c36e000: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:12.908051 7639 tablet_bootstrap.cc:492] T bd95a7eff75845c0b49790c9657ef1e0 P 72a601327d9d447aa7957cde6c36e000: Bootstrap starting.
I20240501 07:59:12.931286 7639 log.cc:824] T bd95a7eff75845c0b49790c9657ef1e0 P 72a601327d9d447aa7957cde6c36e000: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:12.947154 5727 consensus_queue.cc:237] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.70, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } }
I20240501 07:59:12.948655 5862 raft_consensus.cc:1261] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Refusing update from remote peer 46c782f15fb643acb2ef72105f1fb04a: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240501 07:59:12.949116 7575 consensus_queue.cc:1035] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20240501 07:59:12.950712 7575 raft_consensus.cc:2932] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [term 1 LEADER]: Committing config change with OpId 1.71: config changed from index -1 to 71, VOTER 12f3f54871994efeb833126452c61da9 (127.0.99.132) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:12.951411 5862 raft_consensus.cc:2932] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index -1 to 71, VOTER 12f3f54871994efeb833126452c61da9 (127.0.99.132) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
W20240501 07:59:12.954738 5667 consensus_peers.cc:485] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a -> Peer 12f3f54871994efeb833126452c61da9 (127.0.99.132:38041): Couldn't send request to peer 12f3f54871994efeb833126452c61da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dad26af7f84e42208fd4461656117035. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:12.954847 5586 catalog_manager.cc:5484] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723 reported cstate change: config changed from index -1 to 71, VOTER 12f3f54871994efeb833126452c61da9 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "46c782f15fb643acb2ef72105f1fb04a" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:13.018687 7674 ts_tablet_manager.cc:902] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9: Initiating tablet copy from peer ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531)
I20240501 07:59:13.021448 7674 tablet_copy_client.cc:308] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38531
I20240501 07:59:13.039427 6013 tablet_copy_service.cc:140] P ec372f1e72b040e780e6bdd4d5409d33: Received BeginTabletCopySession request for tablet 7f4b2166aae54490a36ced8d37caf7b1 from peer 12f3f54871994efeb833126452c61da9 ({username='slave'} at 127.0.99.132:37561)
I20240501 07:59:13.039611 6013 tablet_copy_service.cc:161] P ec372f1e72b040e780e6bdd4d5409d33: Beginning new tablet copy session on tablet 7f4b2166aae54490a36ced8d37caf7b1 from peer 12f3f54871994efeb833126452c61da9 at {username='slave'} at 127.0.99.132:37561: session id = 12f3f54871994efeb833126452c61da9-7f4b2166aae54490a36ced8d37caf7b1
I20240501 07:59:13.041193 6013 tablet_copy_source_session.cc:205] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:13.047282 7674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f4b2166aae54490a36ced8d37caf7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.059006 7674 tablet_copy_client.cc:783] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:13.061740 7674 tablet_copy_client.cc:647] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:13.068714 7639 tablet_bootstrap.cc:492] T bd95a7eff75845c0b49790c9657ef1e0 P 72a601327d9d447aa7957cde6c36e000: Bootstrap replayed 1/1 log segments. Stats: ops{read=58 overwritten=0 applied=58 ignored=0} inserts{seen=743 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:13.069165 7639 tablet_bootstrap.cc:492] T bd95a7eff75845c0b49790c9657ef1e0 P 72a601327d9d447aa7957cde6c36e000: Bootstrap complete.
I20240501 07:59:13.070072 7674 tablet_copy_client.cc:523] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:13.072400 7674 tablet_bootstrap.cc:492] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9: Bootstrap starting.
I20240501 07:59:13.082697 7639 ts_tablet_manager.cc:1367] T bd95a7eff75845c0b49790c9657ef1e0 P 72a601327d9d447aa7957cde6c36e000: Time spent bootstrapping tablet: real 0.175s user 0.069s sys 0.010s
I20240501 07:59:13.084602 5861 consensus_queue.cc:237] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.66, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:13.087249 7674 log.cc:824] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:13.087250 7639 raft_consensus.cc:348] T bd95a7eff75845c0b49790c9657ef1e0 P 72a601327d9d447aa7957cde6c36e000 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:13.087692 7639 raft_consensus.cc:729] T bd95a7eff75845c0b49790c9657ef1e0 P 72a601327d9d447aa7957cde6c36e000 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a601327d9d447aa7957cde6c36e000, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.088271 7639 consensus_queue.cc:260] T bd95a7eff75845c0b49790c9657ef1e0 P 72a601327d9d447aa7957cde6c36e000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 58, Last appended: 1.58, Last appended by leader: 58, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:13.089352 7639 ts_tablet_manager.cc:1398] T bd95a7eff75845c0b49790c9657ef1e0 P 72a601327d9d447aa7957cde6c36e000: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:59:13.089982 5993 raft_consensus.cc:1261] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 1 index: 66. Preceding OpId from leader: term: 1 index: 67. (index mismatch)
I20240501 07:59:13.090566 7478 consensus_queue.cc:1035] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 65, Time since last communication: 0.000s
I20240501 07:59:13.092361 6857 raft_consensus.cc:2932] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Committing config change with OpId 1.67: config changed from index -1 to 67, VOTER 72a601327d9d447aa7957cde6c36e000 (127.0.99.133) added. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:13.092777 5751 tablet_copy_service.cc:342] P 46c782f15fb643acb2ef72105f1fb04a: Request end of tablet copy session 72a601327d9d447aa7957cde6c36e000-bd95a7eff75845c0b49790c9657ef1e0 received from {username='slave'} at 127.0.99.133:49867
I20240501 07:59:13.092893 5751 tablet_copy_service.cc:433] P 46c782f15fb643acb2ef72105f1fb04a: ending tablet copy session 72a601327d9d447aa7957cde6c36e000-bd95a7eff75845c0b49790c9657ef1e0 on tablet bd95a7eff75845c0b49790c9657ef1e0 with peer 72a601327d9d447aa7957cde6c36e000
W20240501 07:59:13.094861 5796 consensus_peers.cc:485] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 -> Peer 72a601327d9d447aa7957cde6c36e000 (127.0.99.133:34267): Couldn't send request to peer 72a601327d9d447aa7957cde6c36e000. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26ad4c1ac47544f586d3f46f5dc9c77e. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:13.098942 5586 catalog_manager.cc:5484] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 reported cstate change: config changed from index -1 to 67, VOTER 72a601327d9d447aa7957cde6c36e000 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:13.100544 5993 raft_consensus.cc:2932] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Committing config change with OpId 1.67: config changed from index -1 to 67, VOTER 72a601327d9d447aa7957cde6c36e000 (127.0.99.133) added. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:13.153728 7674 tablet_bootstrap.cc:492] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=57 overwritten=0 applied=57 ignored=0} inserts{seen=688 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:13.154163 7674 tablet_bootstrap.cc:492] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9: Bootstrap complete.
I20240501 07:59:13.154603 7674 ts_tablet_manager.cc:1367] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9: Time spent bootstrapping tablet: real 0.082s user 0.071s sys 0.000s
I20240501 07:59:13.157089 7674 raft_consensus.cc:348] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } }
I20240501 07:59:13.157518 7674 raft_consensus.cc:729] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f3f54871994efeb833126452c61da9, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.158073 7674 consensus_queue.cc:260] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 57, Last appended: 1.57, Last appended by leader: 57, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } }
I20240501 07:59:13.159090 7674 ts_tablet_manager.cc:1398] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 07:59:13.167975 6013 tablet_copy_service.cc:342] P ec372f1e72b040e780e6bdd4d5409d33: Request end of tablet copy session 12f3f54871994efeb833126452c61da9-7f4b2166aae54490a36ced8d37caf7b1 received from {username='slave'} at 127.0.99.132:37561
I20240501 07:59:13.168107 6013 tablet_copy_service.cc:433] P ec372f1e72b040e780e6bdd4d5409d33: ending tablet copy session 12f3f54871994efeb833126452c61da9-7f4b2166aae54490a36ced8d37caf7b1 on tablet 7f4b2166aae54490a36ced8d37caf7b1 with peer 12f3f54871994efeb833126452c61da9
I20240501 07:59:13.188699 5862 consensus_queue.cc:237] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 1.67, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } }
W20240501 07:59:13.190340 5795 consensus_peers.cc:485] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 -> Peer a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135:45001): Couldn't send request to peer a3af86b0e9a04ecbb15d305c48d7acb4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 114e9265cef4425799ad44e73e053b59. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:13.191460 5731 raft_consensus.cc:1261] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch)
I20240501 07:59:13.195989 7478 consensus_queue.cc:1035] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20240501 07:59:13.201431 7411 raft_consensus.cc:2932] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Committing config change with OpId 1.68: config changed from index -1 to 68, VOTER a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135) added. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } }
I20240501 07:59:13.202848 5727 raft_consensus.cc:2932] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index -1 to 68, VOTER a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135) added. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } }
I20240501 07:59:13.205569 5586 catalog_manager.cc:5484] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 reported cstate change: config changed from index -1 to 68, VOTER a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } }
I20240501 07:59:13.274492 5861 consensus_queue.cc:237] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 1.58, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:13.276113 7724 ts_tablet_manager.cc:902] T 3f187bcb4c494805a8d6f366732b695a P afc8c07c29354a04926027f57b8c6fab: Initiating tablet copy from peer ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531)
I20240501 07:59:13.278146 7724 tablet_copy_client.cc:308] T 3f187bcb4c494805a8d6f366732b695a P afc8c07c29354a04926027f57b8c6fab: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38531
I20240501 07:59:13.281193 5993 raft_consensus.cc:1261] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240501 07:59:13.282581 7473 consensus_queue.cc:1035] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20240501 07:59:13.285567 6013 tablet_copy_service.cc:140] P ec372f1e72b040e780e6bdd4d5409d33: Received BeginTabletCopySession request for tablet 3f187bcb4c494805a8d6f366732b695a from peer afc8c07c29354a04926027f57b8c6fab ({username='slave'} at 127.0.99.134:47733)
I20240501 07:59:13.285696 6013 tablet_copy_service.cc:161] P ec372f1e72b040e780e6bdd4d5409d33: Beginning new tablet copy session on tablet 3f187bcb4c494805a8d6f366732b695a from peer afc8c07c29354a04926027f57b8c6fab at {username='slave'} at 127.0.99.134:47733: session id = afc8c07c29354a04926027f57b8c6fab-3f187bcb4c494805a8d6f366732b695a
I20240501 07:59:13.286751 6013 tablet_copy_source_session.cc:205] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 07:59:13.287480 5795 consensus_peers.cc:485] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 -> Peer afc8c07c29354a04926027f57b8c6fab (127.0.99.134:38017): Couldn't send request to peer afc8c07c29354a04926027f57b8c6fab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d26c0973a2dc4c639bc3a616e28fbfef. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:13.287781 7411 raft_consensus.cc:2932] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Committing config change with OpId 1.59: config changed from index -1 to 59, VOTER afc8c07c29354a04926027f57b8c6fab (127.0.99.134) added. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:13.291462 7724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f187bcb4c494805a8d6f366732b695a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.291623 5588 catalog_manager.cc:5484] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 reported cstate change: config changed from index -1 to 59, VOTER afc8c07c29354a04926027f57b8c6fab (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:13.295079 5992 raft_consensus.cc:2932] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index -1 to 59, VOTER afc8c07c29354a04926027f57b8c6fab (127.0.99.134) added. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:13.296352 7724 tablet_copy_client.cc:783] T 3f187bcb4c494805a8d6f366732b695a P afc8c07c29354a04926027f57b8c6fab: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:13.297060 7724 tablet_copy_client.cc:647] T 3f187bcb4c494805a8d6f366732b695a P afc8c07c29354a04926027f57b8c6fab: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:13.299446 7734 ts_tablet_manager.cc:902] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4: Initiating tablet copy from peer f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187)
I20240501 07:59:13.301591 7734 tablet_copy_client.cc:308] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42187
I20240501 07:59:13.305493 7724 tablet_copy_client.cc:523] T 3f187bcb4c494805a8d6f366732b695a P afc8c07c29354a04926027f57b8c6fab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:13.307379 7724 tablet_bootstrap.cc:492] T 3f187bcb4c494805a8d6f366732b695a P afc8c07c29354a04926027f57b8c6fab: Bootstrap starting.
I20240501 07:59:13.321830 5882 tablet_copy_service.cc:140] P f022e3632e9a428882a6cd16dc398723: Received BeginTabletCopySession request for tablet a07321bfc4c1445a906ec7c1261693e9 from peer a3af86b0e9a04ecbb15d305c48d7acb4 ({username='slave'} at 127.0.99.135:44527)
I20240501 07:59:13.322010 5882 tablet_copy_service.cc:161] P f022e3632e9a428882a6cd16dc398723: Beginning new tablet copy session on tablet a07321bfc4c1445a906ec7c1261693e9 from peer a3af86b0e9a04ecbb15d305c48d7acb4 at {username='slave'} at 127.0.99.135:44527: session id = a3af86b0e9a04ecbb15d305c48d7acb4-a07321bfc4c1445a906ec7c1261693e9
I20240501 07:59:13.323480 5882 tablet_copy_source_session.cc:205] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:13.325906 7734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a07321bfc4c1445a906ec7c1261693e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.329291 7734 tablet_copy_client.cc:783] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:13.329514 7734 tablet_copy_client.cc:647] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:13.334213 7734 tablet_copy_client.cc:523] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:13.337734 7734 tablet_bootstrap.cc:492] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4: Bootstrap starting.
I20240501 07:59:13.363392 7071 raft_consensus.cc:1203] T bd95a7eff75845c0b49790c9657ef1e0 P 72a601327d9d447aa7957cde6c36e000 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.49->[1.50-1.75] Dedup: 1.58->[1.59-1.75]
I20240501 07:59:13.368266 7674 ts_tablet_manager.cc:902] T dad26af7f84e42208fd4461656117035 P 12f3f54871994efeb833126452c61da9: Initiating tablet copy from peer 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797)
I20240501 07:59:13.368405 5861 consensus_queue.cc:237] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 1.53, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:13.368747 7674 tablet_copy_client.cc:308] T dad26af7f84e42208fd4461656117035 P 12f3f54871994efeb833126452c61da9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33797
W20240501 07:59:13.370905 5796 consensus_peers.cc:485] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 -> Peer 72a601327d9d447aa7957cde6c36e000 (127.0.99.133:34267): Couldn't send request to peer 72a601327d9d447aa7957cde6c36e000. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 662d37043f054aa1a562cde8d6c155a9. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:13.373438 5727 raft_consensus.cc:1261] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 1 index: 53. Preceding OpId from leader: term: 1 index: 54. (index mismatch)
I20240501 07:59:13.374027 7473 consensus_queue.cc:1035] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20240501 07:59:13.375115 5751 tablet_copy_service.cc:140] P 46c782f15fb643acb2ef72105f1fb04a: Received BeginTabletCopySession request for tablet dad26af7f84e42208fd4461656117035 from peer 12f3f54871994efeb833126452c61da9 ({username='slave'} at 127.0.99.132:58269)
I20240501 07:59:13.375216 5751 tablet_copy_service.cc:161] P 46c782f15fb643acb2ef72105f1fb04a: Beginning new tablet copy session on tablet dad26af7f84e42208fd4461656117035 from peer 12f3f54871994efeb833126452c61da9 at {username='slave'} at 127.0.99.132:58269: session id = 12f3f54871994efeb833126452c61da9-dad26af7f84e42208fd4461656117035
I20240501 07:59:13.376216 5751 tablet_copy_source_session.cc:205] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:13.377449 7674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad26af7f84e42208fd4461656117035. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.379745 7674 tablet_copy_client.cc:783] T dad26af7f84e42208fd4461656117035 P 12f3f54871994efeb833126452c61da9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:13.379906 7674 tablet_copy_client.cc:647] T dad26af7f84e42208fd4461656117035 P 12f3f54871994efeb833126452c61da9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:13.380549 7473 raft_consensus.cc:2932] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Committing config change with OpId 1.54: config changed from index -1 to 54, VOTER 72a601327d9d447aa7957cde6c36e000 (127.0.99.133) added. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:13.385597 5727 raft_consensus.cc:2932] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Committing config change with OpId 1.54: config changed from index -1 to 54, VOTER 72a601327d9d447aa7957cde6c36e000 (127.0.99.133) added. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:13.385984 5586 catalog_manager.cc:5484] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 reported cstate change: config changed from index -1 to 54, VOTER 72a601327d9d447aa7957cde6c36e000 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:13.417469 7674 tablet_copy_client.cc:523] T dad26af7f84e42208fd4461656117035 P 12f3f54871994efeb833126452c61da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:13.420255 7674 tablet_bootstrap.cc:492] T dad26af7f84e42208fd4461656117035 P 12f3f54871994efeb833126452c61da9: Bootstrap starting.
I20240501 07:59:13.430297 7724 tablet_bootstrap.cc:492] T 3f187bcb4c494805a8d6f366732b695a P afc8c07c29354a04926027f57b8c6fab: Bootstrap replayed 1/1 log segments. Stats: ops{read=71 overwritten=0 applied=71 ignored=0} inserts{seen=702 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:13.430747 7724 tablet_bootstrap.cc:492] T 3f187bcb4c494805a8d6f366732b695a P afc8c07c29354a04926027f57b8c6fab: Bootstrap complete.
I20240501 07:59:13.430989 7724 ts_tablet_manager.cc:1367] T 3f187bcb4c494805a8d6f366732b695a P afc8c07c29354a04926027f57b8c6fab: Time spent bootstrapping tablet: real 0.124s user 0.076s sys 0.004s
I20240501 07:59:13.431792 7724 raft_consensus.cc:348] T 3f187bcb4c494805a8d6f366732b695a P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:13.431990 7724 raft_consensus.cc:729] T 3f187bcb4c494805a8d6f366732b695a P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc8c07c29354a04926027f57b8c6fab, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.432204 7724 consensus_queue.cc:260] T 3f187bcb4c494805a8d6f366732b695a P afc8c07c29354a04926027f57b8c6fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 71, Last appended: 1.71, Last appended by leader: 71, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:13.432644 7724 ts_tablet_manager.cc:1398] T 3f187bcb4c494805a8d6f366732b695a P afc8c07c29354a04926027f57b8c6fab: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:13.435619 6013 tablet_copy_service.cc:342] P ec372f1e72b040e780e6bdd4d5409d33: Request end of tablet copy session afc8c07c29354a04926027f57b8c6fab-3f187bcb4c494805a8d6f366732b695a received from {username='slave'} at 127.0.99.134:47733
I20240501 07:59:13.435716 6013 tablet_copy_service.cc:433] P ec372f1e72b040e780e6bdd4d5409d33: ending tablet copy session afc8c07c29354a04926027f57b8c6fab-3f187bcb4c494805a8d6f366732b695a on tablet 3f187bcb4c494805a8d6f366732b695a with peer afc8c07c29354a04926027f57b8c6fab
I20240501 07:59:13.477555 5992 consensus_queue.cc:237] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 1.63, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } }
W20240501 07:59:13.479427 5929 consensus_peers.cc:485] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 -> Peer 12f3f54871994efeb833126452c61da9 (127.0.99.132:38041): Couldn't send request to peer 12f3f54871994efeb833126452c61da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a577537e9294fd5bd8507232d1d1a76. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:13.509642 5730 raft_consensus.cc:1261] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240501 07:59:13.510452 7534 consensus_queue.cc:1035] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20240501 07:59:13.511170 7734 tablet_bootstrap.cc:492] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=60 overwritten=0 applied=60 ignored=0} inserts{seen=732 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:13.511485 7734 tablet_bootstrap.cc:492] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4: Bootstrap complete.
I20240501 07:59:13.511725 7734 ts_tablet_manager.cc:1367] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4: Time spent bootstrapping tablet: real 0.174s user 0.065s sys 0.008s
I20240501 07:59:13.512447 7734 raft_consensus.cc:348] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } }
I20240501 07:59:13.512568 7734 raft_consensus.cc:729] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3af86b0e9a04ecbb15d305c48d7acb4, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.512737 7734 consensus_queue.cc:260] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 60, Last appended: 1.60, Last appended by leader: 60, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } }
I20240501 07:59:13.515014 7734 ts_tablet_manager.cc:1398] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:13.517546 5882 tablet_copy_service.cc:342] P f022e3632e9a428882a6cd16dc398723: Request end of tablet copy session a3af86b0e9a04ecbb15d305c48d7acb4-a07321bfc4c1445a906ec7c1261693e9 received from {username='slave'} at 127.0.99.135:44527
I20240501 07:59:13.517669 5882 tablet_copy_service.cc:433] P f022e3632e9a428882a6cd16dc398723: ending tablet copy session a3af86b0e9a04ecbb15d305c48d7acb4-a07321bfc4c1445a906ec7c1261693e9 on tablet a07321bfc4c1445a906ec7c1261693e9 with peer a3af86b0e9a04ecbb15d305c48d7acb4
I20240501 07:59:13.532122 7674 tablet_bootstrap.cc:492] T dad26af7f84e42208fd4461656117035 P 12f3f54871994efeb833126452c61da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=86 overwritten=0 applied=86 ignored=0} inserts{seen=771 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:13.532759 7674 tablet_bootstrap.cc:492] T dad26af7f84e42208fd4461656117035 P 12f3f54871994efeb833126452c61da9: Bootstrap complete.
I20240501 07:59:13.532596 7534 raft_consensus.cc:2932] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Committing config change with OpId 1.64: config changed from index -1 to 64, VOTER 12f3f54871994efeb833126452c61da9 (127.0.99.132) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:13.532974 7674 ts_tablet_manager.cc:1367] T dad26af7f84e42208fd4461656117035 P 12f3f54871994efeb833126452c61da9: Time spent bootstrapping tablet: real 0.113s user 0.083s sys 0.003s
I20240501 07:59:13.533600 7674 raft_consensus.cc:348] T dad26af7f84e42208fd4461656117035 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } }
I20240501 07:59:13.533713 7674 raft_consensus.cc:729] T dad26af7f84e42208fd4461656117035 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f3f54871994efeb833126452c61da9, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.533854 7674 consensus_queue.cc:260] T dad26af7f84e42208fd4461656117035 P 12f3f54871994efeb833126452c61da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 86, Last appended: 1.86, Last appended by leader: 86, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } }
I20240501 07:59:13.537369 7674 ts_tablet_manager.cc:1398] T dad26af7f84e42208fd4461656117035 P 12f3f54871994efeb833126452c61da9: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:13.538988 5588 catalog_manager.cc:5484] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 reported cstate change: config changed from index -1 to 64, VOTER 12f3f54871994efeb833126452c61da9 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:13.539908 5751 tablet_copy_service.cc:342] P 46c782f15fb643acb2ef72105f1fb04a: Request end of tablet copy session 12f3f54871994efeb833126452c61da9-dad26af7f84e42208fd4461656117035 received from {username='slave'} at 127.0.99.132:58269
I20240501 07:59:13.539990 5751 tablet_copy_service.cc:433] P 46c782f15fb643acb2ef72105f1fb04a: ending tablet copy session 12f3f54871994efeb833126452c61da9-dad26af7f84e42208fd4461656117035 on tablet dad26af7f84e42208fd4461656117035 with peer 12f3f54871994efeb833126452c61da9
I20240501 07:59:13.542701 5729 raft_consensus.cc:2932] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index -1 to 64, VOTER 12f3f54871994efeb833126452c61da9 (127.0.99.132) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:13.558521 6940 raft_consensus.cc:1203] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.39->[1.40-1.61] Dedup: 1.57->[1.58-1.61]
I20240501 07:59:13.559523 7639 ts_tablet_manager.cc:902] T 26ad4c1ac47544f586d3f46f5dc9c77e P 72a601327d9d447aa7957cde6c36e000: Initiating tablet copy from peer f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187)
I20240501 07:59:13.560029 7639 tablet_copy_client.cc:308] T 26ad4c1ac47544f586d3f46f5dc9c77e P 72a601327d9d447aa7957cde6c36e000: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42187
I20240501 07:59:13.612427 5882 tablet_copy_service.cc:140] P f022e3632e9a428882a6cd16dc398723: Received BeginTabletCopySession request for tablet 26ad4c1ac47544f586d3f46f5dc9c77e from peer 72a601327d9d447aa7957cde6c36e000 ({username='slave'} at 127.0.99.133:45825)
I20240501 07:59:13.612555 5882 tablet_copy_service.cc:161] P f022e3632e9a428882a6cd16dc398723: Beginning new tablet copy session on tablet 26ad4c1ac47544f586d3f46f5dc9c77e from peer 72a601327d9d447aa7957cde6c36e000 at {username='slave'} at 127.0.99.133:45825: session id = 72a601327d9d447aa7957cde6c36e000-26ad4c1ac47544f586d3f46f5dc9c77e
I20240501 07:59:13.613165 7734 ts_tablet_manager.cc:902] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4: Initiating tablet copy from peer f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187)
I20240501 07:59:13.613420 5882 tablet_copy_source_session.cc:205] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:13.613799 7734 tablet_copy_client.cc:308] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42187
I20240501 07:59:13.615224 7639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ad4c1ac47544f586d3f46f5dc9c77e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.617595 7639 tablet_copy_client.cc:783] T 26ad4c1ac47544f586d3f46f5dc9c77e P 72a601327d9d447aa7957cde6c36e000: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:13.617774 7639 tablet_copy_client.cc:647] T 26ad4c1ac47544f586d3f46f5dc9c77e P 72a601327d9d447aa7957cde6c36e000: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:13.619428 5882 tablet_copy_service.cc:140] P f022e3632e9a428882a6cd16dc398723: Received BeginTabletCopySession request for tablet 114e9265cef4425799ad44e73e053b59 from peer a3af86b0e9a04ecbb15d305c48d7acb4 ({username='slave'} at 127.0.99.135:44527)
I20240501 07:59:13.619532 5882 tablet_copy_service.cc:161] P f022e3632e9a428882a6cd16dc398723: Beginning new tablet copy session on tablet 114e9265cef4425799ad44e73e053b59 from peer a3af86b0e9a04ecbb15d305c48d7acb4 at {username='slave'} at 127.0.99.135:44527: session id = a3af86b0e9a04ecbb15d305c48d7acb4-114e9265cef4425799ad44e73e053b59
I20240501 07:59:13.620404 5882 tablet_copy_source_session.cc:205] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:13.622042 7734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 114e9265cef4425799ad44e73e053b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.623188 7639 tablet_copy_client.cc:523] T 26ad4c1ac47544f586d3f46f5dc9c77e P 72a601327d9d447aa7957cde6c36e000: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:13.624141 7734 tablet_copy_client.cc:783] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:13.624572 7639 tablet_bootstrap.cc:492] T 26ad4c1ac47544f586d3f46f5dc9c77e P 72a601327d9d447aa7957cde6c36e000: Bootstrap starting.
I20240501 07:59:13.626058 7734 tablet_copy_client.cc:647] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:13.628981 7734 tablet_copy_client.cc:523] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:13.634227 7734 tablet_bootstrap.cc:492] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4: Bootstrap starting.
I20240501 07:59:13.666889 5861 consensus_queue.cc:237] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.70, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } }
W20240501 07:59:13.669219 5795 consensus_peers.cc:485] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 -> Peer a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135:45001): Couldn't send request to peer a3af86b0e9a04ecbb15d305c48d7acb4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 877dfd1e30ee4e12b16674d96d57c48e. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:13.669781 5729 raft_consensus.cc:1261] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240501 07:59:13.670190 7473 consensus_queue.cc:1035] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20240501 07:59:13.673725 7473 raft_consensus.cc:2932] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Committing config change with OpId 1.71: config changed from index -1 to 71, VOTER a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } }
I20240501 07:59:13.677464 5588 catalog_manager.cc:5484] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 reported cstate change: config changed from index -1 to 71, VOTER a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } }
I20240501 07:59:13.686628 5729 raft_consensus.cc:2932] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index -1 to 71, VOTER a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } }
I20240501 07:59:13.710970 7336 raft_consensus.cc:1203] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.66] Dedup: 1.60->[1.61-1.66]
I20240501 07:59:13.805088 7734 tablet_bootstrap.cc:492] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=84 overwritten=0 applied=83 ignored=0} inserts{seen=768 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:59:13.805403 7734 tablet_bootstrap.cc:492] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4: Bootstrap complete.
I20240501 07:59:13.805568 7734 ts_tablet_manager.cc:1367] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4: Time spent bootstrapping tablet: real 0.171s user 0.067s sys 0.013s
I20240501 07:59:13.806005 7734 raft_consensus.cc:348] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } }
I20240501 07:59:13.806288 7734 raft_consensus.cc:729] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3af86b0e9a04ecbb15d305c48d7acb4, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.806488 7734 consensus_queue.cc:260] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 83, Last appended: 1.84, Last appended by leader: 84, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } }
I20240501 07:59:13.806783 7734 ts_tablet_manager.cc:1398] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 07:59:13.810639 5882 tablet_copy_service.cc:342] P f022e3632e9a428882a6cd16dc398723: Request end of tablet copy session a3af86b0e9a04ecbb15d305c48d7acb4-114e9265cef4425799ad44e73e053b59 received from {username='slave'} at 127.0.99.135:44527
I20240501 07:59:13.810765 5882 tablet_copy_service.cc:433] P f022e3632e9a428882a6cd16dc398723: ending tablet copy session a3af86b0e9a04ecbb15d305c48d7acb4-114e9265cef4425799ad44e73e053b59 on tablet 114e9265cef4425799ad44e73e053b59 with peer a3af86b0e9a04ecbb15d305c48d7acb4
I20240501 07:59:13.825714 7639 tablet_bootstrap.cc:492] T 26ad4c1ac47544f586d3f46f5dc9c77e P 72a601327d9d447aa7957cde6c36e000: Bootstrap replayed 1/1 log segments. Stats: ops{read=81 overwritten=0 applied=81 ignored=0} inserts{seen=753 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:13.826050 7639 tablet_bootstrap.cc:492] T 26ad4c1ac47544f586d3f46f5dc9c77e P 72a601327d9d447aa7957cde6c36e000: Bootstrap complete.
I20240501 07:59:13.826258 7639 ts_tablet_manager.cc:1367] T 26ad4c1ac47544f586d3f46f5dc9c77e P 72a601327d9d447aa7957cde6c36e000: Time spent bootstrapping tablet: real 0.202s user 0.073s sys 0.006s
I20240501 07:59:13.826880 7639 raft_consensus.cc:348] T 26ad4c1ac47544f586d3f46f5dc9c77e P 72a601327d9d447aa7957cde6c36e000 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:13.827008 7639 raft_consensus.cc:729] T 26ad4c1ac47544f586d3f46f5dc9c77e P 72a601327d9d447aa7957cde6c36e000 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a601327d9d447aa7957cde6c36e000, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.827271 7639 consensus_queue.cc:260] T 26ad4c1ac47544f586d3f46f5dc9c77e P 72a601327d9d447aa7957cde6c36e000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 81, Last appended: 1.81, Last appended by leader: 81, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:13.827595 7639 ts_tablet_manager.cc:1398] T 26ad4c1ac47544f586d3f46f5dc9c77e P 72a601327d9d447aa7957cde6c36e000: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240501 07:59:13.828133 5882 tablet_copy_service.cc:342] P f022e3632e9a428882a6cd16dc398723: Request end of tablet copy session 72a601327d9d447aa7957cde6c36e000-26ad4c1ac47544f586d3f46f5dc9c77e received from {username='slave'} at 127.0.99.133:45825
I20240501 07:59:13.828212 5882 tablet_copy_service.cc:433] P f022e3632e9a428882a6cd16dc398723: ending tablet copy session 72a601327d9d447aa7957cde6c36e000-26ad4c1ac47544f586d3f46f5dc9c77e on tablet 26ad4c1ac47544f586d3f46f5dc9c77e with peer 72a601327d9d447aa7957cde6c36e000
I20240501 07:59:13.837324 7203 raft_consensus.cc:1203] T 3f187bcb4c494805a8d6f366732b695a P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.54->[1.55-1.84] Dedup: 1.71->[1.72-1.84]
I20240501 07:59:13.855338 5992 consensus_queue.cc:237] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 1.85, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
W20240501 07:59:13.857014 5926 consensus_peers.cc:485] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 -> Peer afc8c07c29354a04926027f57b8c6fab (127.0.99.134:38017): Couldn't send request to peer afc8c07c29354a04926027f57b8c6fab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0776eb886275492285da625d69e77e6b. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:13.858268 7639 ts_tablet_manager.cc:902] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000: Initiating tablet copy from peer f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187)
I20240501 07:59:13.858767 7639 tablet_copy_client.cc:308] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42187
I20240501 07:59:13.859213 5882 tablet_copy_service.cc:140] P f022e3632e9a428882a6cd16dc398723: Received BeginTabletCopySession request for tablet 662d37043f054aa1a562cde8d6c155a9 from peer 72a601327d9d447aa7957cde6c36e000 ({username='slave'} at 127.0.99.133:45825)
I20240501 07:59:13.859326 5882 tablet_copy_service.cc:161] P f022e3632e9a428882a6cd16dc398723: Beginning new tablet copy session on tablet 662d37043f054aa1a562cde8d6c155a9 from peer 72a601327d9d447aa7957cde6c36e000 at {username='slave'} at 127.0.99.133:45825: session id = 72a601327d9d447aa7957cde6c36e000-662d37043f054aa1a562cde8d6c155a9
I20240501 07:59:13.860229 5882 tablet_copy_source_session.cc:205] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:13.862372 7639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 662d37043f054aa1a562cde8d6c155a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.864718 7639 tablet_copy_client.cc:783] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:13.866245 5731 raft_consensus.cc:1261] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240501 07:59:13.868140 7422 consensus_queue.cc:1035] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 84, Time since last communication: 0.000s
I20240501 07:59:13.869061 7639 tablet_copy_client.cc:647] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:13.872519 7449 raft_consensus.cc:2932] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Committing config change with OpId 1.86: config changed from index -1 to 86, VOTER afc8c07c29354a04926027f57b8c6fab (127.0.99.134) added. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:13.876001 7639 tablet_copy_client.cc:523] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:13.877410 5731 raft_consensus.cc:2932] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index -1 to 86, VOTER afc8c07c29354a04926027f57b8c6fab (127.0.99.134) added. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:13.877776 7639 tablet_bootstrap.cc:492] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000: Bootstrap starting.
I20240501 07:59:13.878837 5586 catalog_manager.cc:5484] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 reported cstate change: config changed from index -1 to 86, VOTER afc8c07c29354a04926027f57b8c6fab (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:13.901777 7724 ts_tablet_manager.cc:902] T d26c0973a2dc4c639bc3a616e28fbfef P afc8c07c29354a04926027f57b8c6fab: Initiating tablet copy from peer f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187)
I20240501 07:59:13.902343 7724 tablet_copy_client.cc:308] T d26c0973a2dc4c639bc3a616e28fbfef P afc8c07c29354a04926027f57b8c6fab: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42187
I20240501 07:59:13.914374 5882 tablet_copy_service.cc:140] P f022e3632e9a428882a6cd16dc398723: Received BeginTabletCopySession request for tablet d26c0973a2dc4c639bc3a616e28fbfef from peer afc8c07c29354a04926027f57b8c6fab ({username='slave'} at 127.0.99.134:41081)
I20240501 07:59:13.914532 5882 tablet_copy_service.cc:161] P f022e3632e9a428882a6cd16dc398723: Beginning new tablet copy session on tablet d26c0973a2dc4c639bc3a616e28fbfef from peer afc8c07c29354a04926027f57b8c6fab at {username='slave'} at 127.0.99.134:41081: session id = afc8c07c29354a04926027f57b8c6fab-d26c0973a2dc4c639bc3a616e28fbfef
I20240501 07:59:13.915436 5882 tablet_copy_source_session.cc:205] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:13.916906 7724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d26c0973a2dc4c639bc3a616e28fbfef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.919310 7724 tablet_copy_client.cc:783] T d26c0973a2dc4c639bc3a616e28fbfef P afc8c07c29354a04926027f57b8c6fab: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:13.919486 7724 tablet_copy_client.cc:647] T d26c0973a2dc4c639bc3a616e28fbfef P afc8c07c29354a04926027f57b8c6fab: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:13.925997 7724 tablet_copy_client.cc:523] T d26c0973a2dc4c639bc3a616e28fbfef P afc8c07c29354a04926027f57b8c6fab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:13.927584 7724 tablet_bootstrap.cc:492] T d26c0973a2dc4c639bc3a616e28fbfef P afc8c07c29354a04926027f57b8c6fab: Bootstrap starting.
I20240501 07:59:13.958611 6940 raft_consensus.cc:1203] T dad26af7f84e42208fd4461656117035 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.70->[1.71-1.104] Dedup: 1.86->[1.87-1.104]
I20240501 07:59:13.959919 7639 tablet_bootstrap.cc:492] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000: Bootstrap replayed 1/1 log segments. Stats: ops{read=61 overwritten=0 applied=61 ignored=0} inserts{seen=746 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:13.960309 7639 tablet_bootstrap.cc:492] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000: Bootstrap complete.
I20240501 07:59:13.960551 7639 ts_tablet_manager.cc:1367] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000: Time spent bootstrapping tablet: real 0.083s user 0.057s sys 0.008s
I20240501 07:59:13.968631 7639 raft_consensus.cc:348] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:13.969677 7639 raft_consensus.cc:729] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a601327d9d447aa7957cde6c36e000, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.969930 7639 consensus_queue.cc:260] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 61, Last appended: 1.61, Last appended by leader: 61, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:13.970292 7639 ts_tablet_manager.cc:1398] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 07:59:13.971314 5882 tablet_copy_service.cc:342] P f022e3632e9a428882a6cd16dc398723: Request end of tablet copy session 72a601327d9d447aa7957cde6c36e000-662d37043f054aa1a562cde8d6c155a9 received from {username='slave'} at 127.0.99.133:45825
I20240501 07:59:13.971411 5882 tablet_copy_service.cc:433] P f022e3632e9a428882a6cd16dc398723: ending tablet copy session 72a601327d9d447aa7957cde6c36e000-662d37043f054aa1a562cde8d6c155a9 on tablet 662d37043f054aa1a562cde8d6c155a9 with peer 72a601327d9d447aa7957cde6c36e000
I20240501 07:59:13.999300 7674 ts_tablet_manager.cc:902] T 2a577537e9294fd5bd8507232d1d1a76 P 12f3f54871994efeb833126452c61da9: Initiating tablet copy from peer ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531)
I20240501 07:59:13.999788 7674 tablet_copy_client.cc:308] T 2a577537e9294fd5bd8507232d1d1a76 P 12f3f54871994efeb833126452c61da9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38531
I20240501 07:59:14.004252 6013 tablet_copy_service.cc:140] P ec372f1e72b040e780e6bdd4d5409d33: Received BeginTabletCopySession request for tablet 2a577537e9294fd5bd8507232d1d1a76 from peer 12f3f54871994efeb833126452c61da9 ({username='slave'} at 127.0.99.132:37561)
I20240501 07:59:14.004384 6013 tablet_copy_service.cc:161] P ec372f1e72b040e780e6bdd4d5409d33: Beginning new tablet copy session on tablet 2a577537e9294fd5bd8507232d1d1a76 from peer 12f3f54871994efeb833126452c61da9 at {username='slave'} at 127.0.99.132:37561: session id = 12f3f54871994efeb833126452c61da9-2a577537e9294fd5bd8507232d1d1a76
I20240501 07:59:14.005439 6013 tablet_copy_source_session.cc:205] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:14.006537 7674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a577537e9294fd5bd8507232d1d1a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.009092 7674 tablet_copy_client.cc:783] T 2a577537e9294fd5bd8507232d1d1a76 P 12f3f54871994efeb833126452c61da9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:14.009276 7674 tablet_copy_client.cc:647] T 2a577537e9294fd5bd8507232d1d1a76 P 12f3f54871994efeb833126452c61da9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:14.027913 7674 tablet_copy_client.cc:523] T 2a577537e9294fd5bd8507232d1d1a76 P 12f3f54871994efeb833126452c61da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:14.029546 7674 tablet_bootstrap.cc:492] T 2a577537e9294fd5bd8507232d1d1a76 P 12f3f54871994efeb833126452c61da9: Bootstrap starting.
I20240501 07:59:14.048795 7724 tablet_bootstrap.cc:492] T d26c0973a2dc4c639bc3a616e28fbfef P afc8c07c29354a04926027f57b8c6fab: Bootstrap replayed 1/1 log segments. Stats: ops{read=72 overwritten=0 applied=72 ignored=0} inserts{seen=744 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:14.049368 7724 tablet_bootstrap.cc:492] T d26c0973a2dc4c639bc3a616e28fbfef P afc8c07c29354a04926027f57b8c6fab: Bootstrap complete.
I20240501 07:59:14.049597 7724 ts_tablet_manager.cc:1367] T d26c0973a2dc4c639bc3a616e28fbfef P afc8c07c29354a04926027f57b8c6fab: Time spent bootstrapping tablet: real 0.122s user 0.068s sys 0.008s
I20240501 07:59:14.050158 7724 raft_consensus.cc:348] T d26c0973a2dc4c639bc3a616e28fbfef P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:14.050290 7724 raft_consensus.cc:729] T d26c0973a2dc4c639bc3a616e28fbfef P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc8c07c29354a04926027f57b8c6fab, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.050488 7724 consensus_queue.cc:260] T d26c0973a2dc4c639bc3a616e28fbfef P afc8c07c29354a04926027f57b8c6fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 72, Last appended: 1.72, Last appended by leader: 72, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:14.051506 7724 ts_tablet_manager.cc:1398] T d26c0973a2dc4c639bc3a616e28fbfef P afc8c07c29354a04926027f57b8c6fab: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:14.052071 5882 tablet_copy_service.cc:342] P f022e3632e9a428882a6cd16dc398723: Request end of tablet copy session afc8c07c29354a04926027f57b8c6fab-d26c0973a2dc4c639bc3a616e28fbfef received from {username='slave'} at 127.0.99.134:41081
I20240501 07:59:14.052160 5882 tablet_copy_service.cc:433] P f022e3632e9a428882a6cd16dc398723: ending tablet copy session afc8c07c29354a04926027f57b8c6fab-d26c0973a2dc4c639bc3a616e28fbfef on tablet d26c0973a2dc4c639bc3a616e28fbfef with peer afc8c07c29354a04926027f57b8c6fab
I20240501 07:59:14.061918 5992 consensus_queue.cc:237] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 1.90, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:14.070966 5862 raft_consensus.cc:1261] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240501 07:59:14.071422 6611 consensus_queue.cc:1035] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20240501 07:59:14.076169 6611 raft_consensus.cc:2932] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Committing config change with OpId 1.91: config changed from index -1 to 91, VOTER 72a601327d9d447aa7957cde6c36e000 (127.0.99.133) added. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:14.079084 5588 catalog_manager.cc:5484] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 reported cstate change: config changed from index -1 to 91, VOTER 72a601327d9d447aa7957cde6c36e000 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:14.081808 5861 raft_consensus.cc:2932] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index -1 to 91, VOTER 72a601327d9d447aa7957cde6c36e000 (127.0.99.133) added. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
W20240501 07:59:14.102556 5927 consensus_peers.cc:485] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 -> Peer 72a601327d9d447aa7957cde6c36e000 (127.0.99.133:34267): Couldn't send request to peer 72a601327d9d447aa7957cde6c36e000. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6cb1009e4fb4fb0a8e7d55c2c24ff65. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:14.117225 7071 raft_consensus.cc:1203] T 26ad4c1ac47544f586d3f46f5dc9c77e P 72a601327d9d447aa7957cde6c36e000 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.66->[1.67-1.92] Dedup: 1.81->[1.82-1.92]
I20240501 07:59:14.151227 7674 tablet_bootstrap.cc:492] T 2a577537e9294fd5bd8507232d1d1a76 P 12f3f54871994efeb833126452c61da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=77 overwritten=0 applied=76 ignored=0} inserts{seen=707 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:59:14.151643 7674 tablet_bootstrap.cc:492] T 2a577537e9294fd5bd8507232d1d1a76 P 12f3f54871994efeb833126452c61da9: Bootstrap complete.
I20240501 07:59:14.151837 7674 ts_tablet_manager.cc:1367] T 2a577537e9294fd5bd8507232d1d1a76 P 12f3f54871994efeb833126452c61da9: Time spent bootstrapping tablet: real 0.122s user 0.076s sys 0.001s
I20240501 07:59:14.152294 7674 raft_consensus.cc:348] T 2a577537e9294fd5bd8507232d1d1a76 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } }
I20240501 07:59:14.152902 7674 raft_consensus.cc:729] T 2a577537e9294fd5bd8507232d1d1a76 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f3f54871994efeb833126452c61da9, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.153054 7674 consensus_queue.cc:260] T 2a577537e9294fd5bd8507232d1d1a76 P 12f3f54871994efeb833126452c61da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 76, Last appended: 1.77, Last appended by leader: 77, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } }
I20240501 07:59:14.155156 7674 ts_tablet_manager.cc:1398] T 2a577537e9294fd5bd8507232d1d1a76 P 12f3f54871994efeb833126452c61da9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:14.158963 6013 tablet_copy_service.cc:342] P ec372f1e72b040e780e6bdd4d5409d33: Request end of tablet copy session 12f3f54871994efeb833126452c61da9-2a577537e9294fd5bd8507232d1d1a76 received from {username='slave'} at 127.0.99.132:37561
I20240501 07:59:14.159078 6013 tablet_copy_service.cc:433] P ec372f1e72b040e780e6bdd4d5409d33: ending tablet copy session 12f3f54871994efeb833126452c61da9-2a577537e9294fd5bd8507232d1d1a76 on tablet 2a577537e9294fd5bd8507232d1d1a76 with peer 12f3f54871994efeb833126452c61da9
I20240501 07:59:14.224568 5991 consensus_queue.cc:237] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 1.94, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } }
I20240501 07:59:14.226269 5861 raft_consensus.cc:1261] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 1 index: 94. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240501 07:59:14.226919 7449 consensus_queue.cc:1035] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20240501 07:59:14.228632 7336 raft_consensus.cc:1203] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.67->[1.68-1.105] Dedup: 1.84->[1.85-1.105]
I20240501 07:59:14.231631 6611 raft_consensus.cc:2932] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Committing config change with OpId 1.95: config changed from index -1 to 95, VOTER 12f3f54871994efeb833126452c61da9 (127.0.99.132) added. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:14.235718 5588 catalog_manager.cc:5484] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 reported cstate change: config changed from index -1 to 95, VOTER 12f3f54871994efeb833126452c61da9 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
W20240501 07:59:14.236157 5929 consensus_peers.cc:485] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 -> Peer 12f3f54871994efeb833126452c61da9 (127.0.99.132:38041): Couldn't send request to peer 12f3f54871994efeb833126452c61da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65e9dbbdb1fa459fabc258f9e6878f00. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:14.241001 5861 raft_consensus.cc:2932] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Committing config change with OpId 1.95: config changed from index -1 to 95, VOTER 12f3f54871994efeb833126452c61da9 (127.0.99.132) added. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:14.258206 7071 raft_consensus.cc:1203] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.53->[1.54-1.70] Dedup: 1.61->[1.62-1.70]
I20240501 07:59:14.284204 7734 ts_tablet_manager.cc:902] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4: Initiating tablet copy from peer f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187)
I20240501 07:59:14.284669 7734 tablet_copy_client.cc:308] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42187
I20240501 07:59:14.293082 5882 tablet_copy_service.cc:140] P f022e3632e9a428882a6cd16dc398723: Received BeginTabletCopySession request for tablet 877dfd1e30ee4e12b16674d96d57c48e from peer a3af86b0e9a04ecbb15d305c48d7acb4 ({username='slave'} at 127.0.99.135:44527)
I20240501 07:59:14.293215 5882 tablet_copy_service.cc:161] P f022e3632e9a428882a6cd16dc398723: Beginning new tablet copy session on tablet 877dfd1e30ee4e12b16674d96d57c48e from peer a3af86b0e9a04ecbb15d305c48d7acb4 at {username='slave'} at 127.0.99.135:44527: session id = a3af86b0e9a04ecbb15d305c48d7acb4-877dfd1e30ee4e12b16674d96d57c48e
I20240501 07:59:14.294166 5882 tablet_copy_source_session.cc:205] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:14.297094 7734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 877dfd1e30ee4e12b16674d96d57c48e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.299433 7734 tablet_copy_client.cc:783] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:14.299605 7734 tablet_copy_client.cc:647] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:14.308780 7724 ts_tablet_manager.cc:902] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab: Initiating tablet copy from peer ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531)
I20240501 07:59:14.309293 7724 tablet_copy_client.cc:308] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38531
I20240501 07:59:14.309878 6013 tablet_copy_service.cc:140] P ec372f1e72b040e780e6bdd4d5409d33: Received BeginTabletCopySession request for tablet 0776eb886275492285da625d69e77e6b from peer afc8c07c29354a04926027f57b8c6fab ({username='slave'} at 127.0.99.134:47733)
I20240501 07:59:14.309975 6013 tablet_copy_service.cc:161] P ec372f1e72b040e780e6bdd4d5409d33: Beginning new tablet copy session on tablet 0776eb886275492285da625d69e77e6b from peer afc8c07c29354a04926027f57b8c6fab at {username='slave'} at 127.0.99.134:47733: session id = afc8c07c29354a04926027f57b8c6fab-0776eb886275492285da625d69e77e6b
I20240501 07:59:14.310966 6013 tablet_copy_source_session.cc:205] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:14.314138 7724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0776eb886275492285da625d69e77e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.316550 7724 tablet_copy_client.cc:783] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:14.316715 7724 tablet_copy_client.cc:647] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:14.324848 7734 tablet_copy_client.cc:523] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:14.325289 7202 raft_consensus.cc:1203] T d26c0973a2dc4c639bc3a616e28fbfef P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.58->[1.59-1.87] Dedup: 1.72->[1.73-1.87]
I20240501 07:59:14.326391 7734 tablet_bootstrap.cc:492] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4: Bootstrap starting.
I20240501 07:59:14.353423 7724 tablet_copy_client.cc:523] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:14.355669 7724 tablet_bootstrap.cc:492] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab: Bootstrap starting.
I20240501 07:59:14.486403 7734 tablet_bootstrap.cc:492] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=84 overwritten=0 applied=84 ignored=0} inserts{seen=769 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:14.486758 7734 tablet_bootstrap.cc:492] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4: Bootstrap complete.
I20240501 07:59:14.487661 7734 ts_tablet_manager.cc:1367] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4: Time spent bootstrapping tablet: real 0.161s user 0.084s sys 0.005s
I20240501 07:59:14.488181 7734 raft_consensus.cc:348] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } }
I20240501 07:59:14.488301 7734 raft_consensus.cc:729] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3af86b0e9a04ecbb15d305c48d7acb4, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.488464 7734 consensus_queue.cc:260] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 84, Last appended: 1.84, Last appended by leader: 84, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } }
I20240501 07:59:14.488761 7734 ts_tablet_manager.cc:1398] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:14.489426 7936 ts_tablet_manager.cc:902] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P 72a601327d9d447aa7957cde6c36e000: Initiating tablet copy from peer ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531)
I20240501 07:59:14.489988 7936 tablet_copy_client.cc:308] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P 72a601327d9d447aa7957cde6c36e000: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38531
I20240501 07:59:14.497262 6013 tablet_copy_service.cc:140] P ec372f1e72b040e780e6bdd4d5409d33: Received BeginTabletCopySession request for tablet a6cb1009e4fb4fb0a8e7d55c2c24ff65 from peer 72a601327d9d447aa7957cde6c36e000 ({username='slave'} at 127.0.99.133:41775)
I20240501 07:59:14.497409 6013 tablet_copy_service.cc:161] P ec372f1e72b040e780e6bdd4d5409d33: Beginning new tablet copy session on tablet a6cb1009e4fb4fb0a8e7d55c2c24ff65 from peer 72a601327d9d447aa7957cde6c36e000 at {username='slave'} at 127.0.99.133:41775: session id = 72a601327d9d447aa7957cde6c36e000-a6cb1009e4fb4fb0a8e7d55c2c24ff65
I20240501 07:59:14.498473 6013 tablet_copy_source_session.cc:205] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:14.500550 7936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6cb1009e4fb4fb0a8e7d55c2c24ff65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.502946 7936 tablet_copy_client.cc:783] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P 72a601327d9d447aa7957cde6c36e000: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:14.503119 7936 tablet_copy_client.cc:647] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P 72a601327d9d447aa7957cde6c36e000: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:14.503787 5882 tablet_copy_service.cc:342] P f022e3632e9a428882a6cd16dc398723: Request end of tablet copy session a3af86b0e9a04ecbb15d305c48d7acb4-877dfd1e30ee4e12b16674d96d57c48e received from {username='slave'} at 127.0.99.135:44527
I20240501 07:59:14.503870 5882 tablet_copy_service.cc:433] P f022e3632e9a428882a6cd16dc398723: ending tablet copy session a3af86b0e9a04ecbb15d305c48d7acb4-877dfd1e30ee4e12b16674d96d57c48e on tablet 877dfd1e30ee4e12b16674d96d57c48e with peer a3af86b0e9a04ecbb15d305c48d7acb4
I20240501 07:59:14.506203 7936 tablet_copy_client.cc:523] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P 72a601327d9d447aa7957cde6c36e000: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:14.507786 7936 tablet_bootstrap.cc:492] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P 72a601327d9d447aa7957cde6c36e000: Bootstrap starting.
I20240501 07:59:14.560612 6940 raft_consensus.cc:1203] T 2a577537e9294fd5bd8507232d1d1a76 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.63->[1.64-1.93] Dedup: 1.77->[1.78-1.93]
I20240501 07:59:14.601452 7051 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:14.605301 7317 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:14.606029 7724 tablet_bootstrap.cc:492] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab: Bootstrap replayed 1/1 log segments. Stats: ops{read=88 overwritten=0 applied=87 ignored=0} inserts{seen=718 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 07:59:14.606412 7724 tablet_bootstrap.cc:492] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab: Bootstrap complete.
I20240501 07:59:14.606637 7724 ts_tablet_manager.cc:1367] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab: Time spent bootstrapping tablet: real 0.251s user 0.080s sys 0.008s
I20240501 07:59:14.607126 7724 raft_consensus.cc:348] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:14.607460 7724 raft_consensus.cc:729] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc8c07c29354a04926027f57b8c6fab, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.619191 7724 consensus_queue.cc:260] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 87, Last appended: 1.88, Last appended by leader: 88, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:14.621755 7724 ts_tablet_manager.cc:1398] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240501 07:59:14.622270 6013 tablet_copy_service.cc:342] P ec372f1e72b040e780e6bdd4d5409d33: Request end of tablet copy session afc8c07c29354a04926027f57b8c6fab-0776eb886275492285da625d69e77e6b received from {username='slave'} at 127.0.99.134:47733
I20240501 07:59:14.622357 6013 tablet_copy_service.cc:433] P ec372f1e72b040e780e6bdd4d5409d33: ending tablet copy session afc8c07c29354a04926027f57b8c6fab-0776eb886275492285da625d69e77e6b on tablet 0776eb886275492285da625d69e77e6b with peer afc8c07c29354a04926027f57b8c6fab
I20240501 07:59:14.638933 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:14.655442 5972 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:14.656314 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:14.705902 7936 tablet_bootstrap.cc:492] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P 72a601327d9d447aa7957cde6c36e000: Bootstrap replayed 1/1 log segments. Stats: ops{read=91 overwritten=0 applied=91 ignored=0} inserts{seen=763 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:14.706241 7936 tablet_bootstrap.cc:492] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P 72a601327d9d447aa7957cde6c36e000: Bootstrap complete.
I20240501 07:59:14.706477 7936 ts_tablet_manager.cc:1367] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P 72a601327d9d447aa7957cde6c36e000: Time spent bootstrapping tablet: real 0.199s user 0.082s sys 0.009s
I20240501 07:59:14.707134 7936 raft_consensus.cc:348] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P 72a601327d9d447aa7957cde6c36e000 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:14.707262 7936 raft_consensus.cc:729] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P 72a601327d9d447aa7957cde6c36e000 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a601327d9d447aa7957cde6c36e000, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.707427 7936 consensus_queue.cc:260] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P 72a601327d9d447aa7957cde6c36e000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 91, Last appended: 1.91, Last appended by leader: 91, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:14.707682 7936 ts_tablet_manager.cc:1398] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P 72a601327d9d447aa7957cde6c36e000: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:14.708083 6013 tablet_copy_service.cc:342] P ec372f1e72b040e780e6bdd4d5409d33: Request end of tablet copy session 72a601327d9d447aa7957cde6c36e000-a6cb1009e4fb4fb0a8e7d55c2c24ff65 received from {username='slave'} at 127.0.99.133:41775
I20240501 07:59:14.708159 6013 tablet_copy_service.cc:433] P ec372f1e72b040e780e6bdd4d5409d33: ending tablet copy session 72a601327d9d447aa7957cde6c36e000-a6cb1009e4fb4fb0a8e7d55c2c24ff65 on tablet a6cb1009e4fb4fb0a8e7d55c2c24ff65 with peer 72a601327d9d447aa7957cde6c36e000
I20240501 07:59:14.713770 5841 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:14.761449 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:14.833807 5861 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "662d37043f054aa1a562cde8d6c155a9"
dest_uuid: "f022e3632e9a428882a6cd16dc398723"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59826
I20240501 07:59:14.834020 5861 raft_consensus.cc:595] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:14.835325 7880 raft_consensus.cc:979] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723: : Instructing follower 72a601327d9d447aa7957cde6c36e000 to start an election
I20240501 07:59:14.835423 7880 raft_consensus.cc:1067] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Signalling peer 72a601327d9d447aa7957cde6c36e000 to start an election
I20240501 07:59:14.835985 7071 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "662d37043f054aa1a562cde8d6c155a9"
dest_uuid: "72a601327d9d447aa7957cde6c36e000"
from {username='slave'} at 127.0.99.130:58221
I20240501 07:59:14.836175 7071 raft_consensus.cc:482] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:14.836308 7071 raft_consensus.cc:3037] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:14.837466 7071 raft_consensus.cc:504] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:14.839146 5726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "662d37043f054aa1a562cde8d6c155a9" candidate_uuid: "72a601327d9d447aa7957cde6c36e000" candidate_term: 2 candidate_status { last_received { term: 1 index: 86 } } ignore_live_leader: true dest_uuid: "46c782f15fb643acb2ef72105f1fb04a"
I20240501 07:59:14.839291 5726 raft_consensus.cc:3037] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:14.839740 7071 leader_election.cc:288] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000 [CANDIDATE]: Term 2 election: Requested vote from peers 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797), f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187)
I20240501 07:59:14.840296 5861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "662d37043f054aa1a562cde8d6c155a9" candidate_uuid: "72a601327d9d447aa7957cde6c36e000" candidate_term: 2 candidate_status { last_received { term: 1 index: 86 } } ignore_live_leader: true dest_uuid: "f022e3632e9a428882a6cd16dc398723"
I20240501 07:59:14.840359 5726 raft_consensus.cc:2454] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72a601327d9d447aa7957cde6c36e000 in term 2.
I20240501 07:59:14.840411 5861 raft_consensus.cc:3032] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:14.840448 5861 raft_consensus.cc:729] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Running, Role: LEADER
I20240501 07:59:14.840595 5861 consensus_queue.cc:260] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 1.86, Last appended by leader: 86, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:14.840785 5861 raft_consensus.cc:3037] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:14.841418 7006 leader_election.cc:302] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46c782f15fb643acb2ef72105f1fb04a, 72a601327d9d447aa7957cde6c36e000; no voters:
I20240501 07:59:14.841667 7692 raft_consensus.cc:2781] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:14.841830 5861 raft_consensus.cc:2454] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72a601327d9d447aa7957cde6c36e000 in term 2.
I20240501 07:59:14.841843 7692 raft_consensus.cc:686] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000 [term 2 LEADER]: Becoming Leader. State: Replica: 72a601327d9d447aa7957cde6c36e000, State: Running, Role: LEADER
I20240501 07:59:14.842172 7692 consensus_queue.cc:237] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 1.86, Last appended by leader: 86, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:14.843369 5731 raft_consensus.cc:1226] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a [term 2 FOLLOWER]: Rejecting Update request from peer f022e3632e9a428882a6cd16dc398723 for earlier term 1. Current term is 2. Ops: [1.86-1.86]
I20240501 07:59:14.845506 5588 catalog_manager.cc:5484] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000 reported cstate change: term changed from 1 to 2, leader changed from f022e3632e9a428882a6cd16dc398723 (127.0.99.130) to 72a601327d9d447aa7957cde6c36e000 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "72a601327d9d447aa7957cde6c36e000" committed_config { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:14.881628 7977 ts_tablet_manager.cc:902] T 65e9dbbdb1fa459fabc258f9e6878f00 P 12f3f54871994efeb833126452c61da9: Initiating tablet copy from peer ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531)
I20240501 07:59:14.882347 7977 tablet_copy_client.cc:308] T 65e9dbbdb1fa459fabc258f9e6878f00 P 12f3f54871994efeb833126452c61da9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38531
I20240501 07:59:14.891994 6013 tablet_copy_service.cc:140] P ec372f1e72b040e780e6bdd4d5409d33: Received BeginTabletCopySession request for tablet 65e9dbbdb1fa459fabc258f9e6878f00 from peer 12f3f54871994efeb833126452c61da9 ({username='slave'} at 127.0.99.132:37561)
I20240501 07:59:14.892127 6013 tablet_copy_service.cc:161] P ec372f1e72b040e780e6bdd4d5409d33: Beginning new tablet copy session on tablet 65e9dbbdb1fa459fabc258f9e6878f00 from peer 12f3f54871994efeb833126452c61da9 at {username='slave'} at 127.0.99.132:37561: session id = 12f3f54871994efeb833126452c61da9-65e9dbbdb1fa459fabc258f9e6878f00
W20240501 07:59:14.892443 5822 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59692: Illegal state: replica f022e3632e9a428882a6cd16dc398723 is not leader of this config: current role FOLLOWER
I20240501 07:59:14.893110 6013 tablet_copy_source_session.cc:205] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:14.900331 7977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65e9dbbdb1fa459fabc258f9e6878f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.900403 7336 raft_consensus.cc:1203] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.70->[1.71-1.100] Dedup: 1.84->[1.85-1.100]
I20240501 07:59:14.911237 7071 raft_consensus.cc:1203] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P 72a601327d9d447aa7957cde6c36e000 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.90->[1.91-1.99] Dedup: 1.91->[1.92-1.99]
I20240501 07:59:14.921548 7203 raft_consensus.cc:1203] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.85->[1.86-1.100] Dedup: 1.88->[1.89-1.100]
I20240501 07:59:14.921810 7977 tablet_copy_client.cc:783] T 65e9dbbdb1fa459fabc258f9e6878f00 P 12f3f54871994efeb833126452c61da9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:14.922116 7977 tablet_copy_client.cc:647] T 65e9dbbdb1fa459fabc258f9e6878f00 P 12f3f54871994efeb833126452c61da9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:14.924207 7977 tablet_copy_client.cc:523] T 65e9dbbdb1fa459fabc258f9e6878f00 P 12f3f54871994efeb833126452c61da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:14.926748 7977 tablet_bootstrap.cc:492] T 65e9dbbdb1fa459fabc258f9e6878f00 P 12f3f54871994efeb833126452c61da9: Bootstrap starting.
I20240501 07:59:15.117345 7977 tablet_bootstrap.cc:492] T 65e9dbbdb1fa459fabc258f9e6878f00 P 12f3f54871994efeb833126452c61da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=110 overwritten=0 applied=110 ignored=0} inserts{seen=782 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:15.117791 7977 tablet_bootstrap.cc:492] T 65e9dbbdb1fa459fabc258f9e6878f00 P 12f3f54871994efeb833126452c61da9: Bootstrap complete.
I20240501 07:59:15.118054 7977 ts_tablet_manager.cc:1367] T 65e9dbbdb1fa459fabc258f9e6878f00 P 12f3f54871994efeb833126452c61da9: Time spent bootstrapping tablet: real 0.191s user 0.094s sys 0.008s
I20240501 07:59:15.118744 7977 raft_consensus.cc:348] T 65e9dbbdb1fa459fabc258f9e6878f00 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } }
I20240501 07:59:15.118969 7977 raft_consensus.cc:729] T 65e9dbbdb1fa459fabc258f9e6878f00 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f3f54871994efeb833126452c61da9, State: Initialized, Role: FOLLOWER
I20240501 07:59:15.119246 7977 consensus_queue.cc:260] T 65e9dbbdb1fa459fabc258f9e6878f00 P 12f3f54871994efeb833126452c61da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 110, Last appended: 1.110, Last appended by leader: 110, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } }
I20240501 07:59:15.119740 7977 ts_tablet_manager.cc:1398] T 65e9dbbdb1fa459fabc258f9e6878f00 P 12f3f54871994efeb833126452c61da9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:15.122934 6013 tablet_copy_service.cc:342] P ec372f1e72b040e780e6bdd4d5409d33: Request end of tablet copy session 12f3f54871994efeb833126452c61da9-65e9dbbdb1fa459fabc258f9e6878f00 received from {username='slave'} at 127.0.99.132:37561
I20240501 07:59:15.123034 6013 tablet_copy_service.cc:433] P ec372f1e72b040e780e6bdd4d5409d33: ending tablet copy session 12f3f54871994efeb833126452c61da9-65e9dbbdb1fa459fabc258f9e6878f00 on tablet 65e9dbbdb1fa459fabc258f9e6878f00 with peer 12f3f54871994efeb833126452c61da9
W20240501 07:59:15.138379 5690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54066: Illegal state: replica 46c782f15fb643acb2ef72105f1fb04a is not leader of this config: current role FOLLOWER
I20240501 07:59:15.160557 5862 raft_consensus.cc:1261] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Refusing update from remote peer 72a601327d9d447aa7957cde6c36e000: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 2 index: 88. (index mismatch)
I20240501 07:59:15.161326 7692 consensus_queue.cc:1035] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240501 07:59:15.162283 5726 raft_consensus.cc:1261] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a [term 2 FOLLOWER]: Refusing update from remote peer 72a601327d9d447aa7957cde6c36e000: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 2 index: 88. (index mismatch)
I20240501 07:59:15.165380 7975 consensus_queue.cc:1035] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 84, Time since last communication: 0.000s
I20240501 07:59:15.166527 7989 mvcc.cc:205] Tried to move back new op lower bound from 7022798254726152192 to 7022798253436567552. Current Snapshot: MvccSnapshot[applied={T|T < 7022798254726152192}]
I20240501 07:59:15.170310 7793 mvcc.cc:205] Tried to move back new op lower bound from 7022798254726152192 to 7022798253436567552. Current Snapshot: MvccSnapshot[applied={T|T < 7022798254726152192}]
I20240501 07:59:15.176966 7969 mvcc.cc:205] Tried to move back new op lower bound from 7022798254726152192 to 7022798253436567552. Current Snapshot: MvccSnapshot[applied={T|T < 7022798254726152192}]
I20240501 07:59:15.235833 6939 raft_consensus.cc:1203] T 65e9dbbdb1fa459fabc258f9e6878f00 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.94->[1.95-1.116] Dedup: 1.110->[1.111-1.116]
I20240501 07:59:15.403147 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:15.440589 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:15.502440 5972 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:15.523916 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:15.525414 7317 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:15.553027 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:15.575978 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:15.869110 7069 consensus_queue.cc:237] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 2.111, Last appended by leader: 86, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:15.879884 5728 raft_consensus.cc:1261] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a [term 2 FOLLOWER]: Refusing update from remote peer 72a601327d9d447aa7957cde6c36e000: Log matching property violated. Preceding OpId in replica: term: 2 index: 111. Preceding OpId from leader: term: 2 index: 112. (index mismatch)
I20240501 07:59:15.881842 7975 consensus_queue.cc:1035] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 110, Time since last communication: 0.000s
I20240501 07:59:15.894932 7975 raft_consensus.cc:2932] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000 [term 2 LEADER]: Committing config change with OpId 2.112: config changed from index 54 to 112, VOTER f022e3632e9a428882a6cd16dc398723 (127.0.99.130) evicted. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:15.899863 5580 catalog_manager.cc:5484] T 662d37043f054aa1a562cde8d6c155a9 P 72a601327d9d447aa7957cde6c36e000 reported cstate change: config changed from index 54 to 112, VOTER f022e3632e9a428882a6cd16dc398723 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "72a601327d9d447aa7957cde6c36e000" committed_config { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:15.897527 5728 raft_consensus.cc:2932] T 662d37043f054aa1a562cde8d6c155a9 P 46c782f15fb643acb2ef72105f1fb04a [term 2 FOLLOWER]: Committing config change with OpId 2.112: config changed from index 54 to 112, VOTER f022e3632e9a428882a6cd16dc398723 (127.0.99.130) evicted. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:15.920238 5841 tablet_service.cc:1514] Processing DeleteTablet for tablet 662d37043f054aa1a562cde8d6c155a9 with delete_type TABLET_DATA_TOMBSTONED (TS f022e3632e9a428882a6cd16dc398723 not found in new config with opid_index 112) from {username='slave'} at 127.0.0.1:54330
I20240501 07:59:15.921115 8091 tablet_replica.cc:330] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723: stopping tablet replica
I20240501 07:59:15.921756 8091 raft_consensus.cc:2229] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:15.921980 8091 pending_rounds.cc:62] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723: Trying to abort 1 pending ops.
I20240501 07:59:15.922051 8091 pending_rounds.cc:68] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723: Aborting op as it isn't in flight: id { term: 2 index: 111 } timestamp: 7022798257622573056 op_type: WRITE_OP write_request { tablet_id: "662d37043f054aa1a562cde8d6c155a9" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\214\230\033Y\010\352Jt\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""12c410ca8c54d073" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022798257592098816 authz_token { token_data: "\010\373\356\307\261\006\"3\n\005slave\022*\n f3b4504f12d4447ba80f01cbb31b8c0a\020\001\030\001 \001(\001" signature: "<redacted>""y\3228%\350P\010-\216\320z\343\203\241\t\363\000\031\203!i\306\2054\0105U\021:\253f\230Y\254\2570\016\364n\026A7\305l\337\002\221\315\253F^\245\033Rt\025\227z\231\353\203\032\321C" signing_key_seq_num: 0 } } request_id { client_id: "b3171811619f48b09877fb8005c460a8" seq_no: 325 first_incomplete_seq_no: 325 attempt_no: 0 }
I20240501 07:59:15.922749 8091 raft_consensus.cc:2258] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:15.924422 8091 ts_tablet_manager.cc:1875] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:15.927425 8091 ts_tablet_manager.cc:1888] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.111
I20240501 07:59:15.927629 8091 log.cc:1196] T 662d37043f054aa1a562cde8d6c155a9 P f022e3632e9a428882a6cd16dc398723: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/662d37043f054aa1a562cde8d6c155a9
I20240501 07:59:15.928656 5573 catalog_manager.cc:4830] TS f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187): tablet 662d37043f054aa1a562cde8d6c155a9 (table rebalance_test_table_2 [id=f3b4504f12d4447ba80f01cbb31b8c0a]) successfully deleted
I20240501 07:59:16.251837 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:16.261617 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:16.282377 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:16.293136 7317 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:16.306263 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:16.307531 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:16.355696 5972 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:16.806128 5862 consensus_queue.cc:237] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 142, Committed index: 142, Last appended: 1.143, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 144 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:16.807241 7202 raft_consensus.cc:1261] T d26c0973a2dc4c639bc3a616e28fbfef P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 1 index: 142. Preceding OpId from leader: term: 1 index: 144. (index mismatch)
I20240501 07:59:16.807725 6857 consensus_queue.cc:1035] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 144, Last known committed idx: 142, Time since last communication: 0.000s
I20240501 07:59:16.809725 6857 raft_consensus.cc:2932] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Committing config change with OpId 1.144: config changed from index 59 to 144, VOTER ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) evicted. New config: { opid_index: 144 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:16.811926 7201 raft_consensus.cc:2932] T d26c0973a2dc4c639bc3a616e28fbfef P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Committing config change with OpId 1.144: config changed from index 59 to 144, VOTER ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) evicted. New config: { opid_index: 144 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:16.814395 7203 raft_consensus.cc:1203] T d26c0973a2dc4c639bc3a616e28fbfef P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.142->[1.143-1.143] Dedup: 1.143->[]
I20240501 07:59:16.817124 5586 catalog_manager.cc:5484] T d26c0973a2dc4c639bc3a616e28fbfef P f022e3632e9a428882a6cd16dc398723 reported cstate change: config changed from index 59 to 144, VOTER ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: 144 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:16.893390 5972 tablet_service.cc:1514] Processing DeleteTablet for tablet d26c0973a2dc4c639bc3a616e28fbfef with delete_type TABLET_DATA_TOMBSTONED (TS ec372f1e72b040e780e6bdd4d5409d33 not found in new config with opid_index 144) from {username='slave'} at 127.0.0.1:38970
I20240501 07:59:16.896692 8169 tablet_replica.cc:330] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33: stopping tablet replica
I20240501 07:59:16.897099 8169 raft_consensus.cc:2229] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:16.897284 8169 pending_rounds.cc:62] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33: Trying to abort 1 pending ops.
I20240501 07:59:16.897375 8169 pending_rounds.cc:68] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33: Aborting op as it isn't in flight: id { term: 1 index: 143 } timestamp: 7022798261469724672 op_type: WRITE_OP write_request { tablet_id: "d26c0973a2dc4c639bc3a616e28fbfef" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\0006\264@s\234\336\n\275\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""a60d377bdb6b4129" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022798261431427072 authz_token { token_data: "\010\373\356\307\261\006\"3\n\005slave\022*\n f3b4504f12d4447ba80f01cbb31b8c0a\020\001\030\001 \001(\001" signature: "<redacted>""y\3228%\350P\010-\216\320z\343\203\241\t\363\000\031\203!i\306\2054\0105U\021:\253f\230Y\254\2570\016\364n\026A7\305l\337\002\221\315\253F^\245\033Rt\025\227z\231\353\203\032\321C" signing_key_seq_num: 0 } } request_id { client_id: "b3171811619f48b09877fb8005c460a8" seq_no: 397 first_incomplete_seq_no: 397 attempt_no: 0 }
I20240501 07:59:16.897754 8169 raft_consensus.cc:2258] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.901342 8169 ts_tablet_manager.cc:1875] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:16.919337 8169 ts_tablet_manager.cc:1888] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.143
I20240501 07:59:16.925107 8169 log.cc:1196] T d26c0973a2dc4c639bc3a616e28fbfef P ec372f1e72b040e780e6bdd4d5409d33: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d26c0973a2dc4c639bc3a616e28fbfef
I20240501 07:59:16.926136 5575 catalog_manager.cc:4830] TS ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531): tablet d26c0973a2dc4c639bc3a616e28fbfef (table rebalance_test_table_2 [id=f3b4504f12d4447ba80f01cbb31b8c0a]) successfully deleted
I20240501 07:59:17.183593 5972 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:17.184134 7317 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:17.217973 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:17.218880 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:17.227733 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:17.238523 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:17.277019 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:17.406625 5859 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "877dfd1e30ee4e12b16674d96d57c48e"
dest_uuid: "f022e3632e9a428882a6cd16dc398723"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59894
I20240501 07:59:17.406788 5859 raft_consensus.cc:595] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:17.412533 8151 raft_consensus.cc:979] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723: : Instructing follower a3af86b0e9a04ecbb15d305c48d7acb4 to start an election
I20240501 07:59:17.412655 8151 raft_consensus.cc:1067] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Signalling peer a3af86b0e9a04ecbb15d305c48d7acb4 to start an election
I20240501 07:59:17.413942 7335 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "877dfd1e30ee4e12b16674d96d57c48e"
dest_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4"
from {username='slave'} at 127.0.99.130:44249
I20240501 07:59:17.414292 7335 raft_consensus.cc:482] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:17.414373 7335 raft_consensus.cc:3037] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:17.415957 7335 raft_consensus.cc:504] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } }
W20240501 07:59:17.417644 5822 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59698: Service unavailable: leader transfer in progress
W20240501 07:59:17.420173 5820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59698: Service unavailable: leader transfer in progress
I20240501 07:59:17.424474 7335 leader_election.cc:288] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [CANDIDATE]: Term 2 election: Requested vote from peers 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797), f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187)
W20240501 07:59:17.425277 5822 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59698: Service unavailable: leader transfer in progress
I20240501 07:59:17.425297 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "877dfd1e30ee4e12b16674d96d57c48e" candidate_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 175 } } ignore_live_leader: true dest_uuid: "f022e3632e9a428882a6cd16dc398723"
I20240501 07:59:17.425516 5859 raft_consensus.cc:3032] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:17.425578 5859 raft_consensus.cc:729] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Running, Role: LEADER
I20240501 07:59:17.425710 5859 consensus_queue.cc:260] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 175, Committed index: 175, Last appended: 1.175, Last appended by leader: 175, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } }
I20240501 07:59:17.425855 5859 raft_consensus.cc:3037] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:17.427013 5859 raft_consensus.cc:2454] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3af86b0e9a04ecbb15d305c48d7acb4 in term 2.
I20240501 07:59:17.427496 7269 leader_election.cc:302] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3af86b0e9a04ecbb15d305c48d7acb4, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:17.427703 8206 raft_consensus.cc:2781] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:17.427778 8206 raft_consensus.cc:686] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [term 2 LEADER]: Becoming Leader. State: Replica: a3af86b0e9a04ecbb15d305c48d7acb4, State: Running, Role: LEADER
I20240501 07:59:17.427960 8206 consensus_queue.cc:237] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 175, Committed index: 175, Last appended: 1.175, Last appended by leader: 175, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } }
I20240501 07:59:17.431185 5580 catalog_manager.cc:5484] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 reported cstate change: term changed from 1 to 2, leader changed from f022e3632e9a428882a6cd16dc398723 (127.0.99.130) to a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } }
W20240501 07:59:17.433341 5822 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59698: Illegal state: replica f022e3632e9a428882a6cd16dc398723 is not leader of this config: current role FOLLOWER
I20240501 07:59:17.453256 5728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "877dfd1e30ee4e12b16674d96d57c48e" candidate_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 175 } } ignore_live_leader: true dest_uuid: "46c782f15fb643acb2ef72105f1fb04a"
I20240501 07:59:17.453403 5728 raft_consensus.cc:3037] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:17.455735 5728 raft_consensus.cc:2454] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3af86b0e9a04ecbb15d305c48d7acb4 in term 2.
W20240501 07:59:17.482817 5690 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54096: Illegal state: replica 46c782f15fb643acb2ef72105f1fb04a is not leader of this config: current role FOLLOWER
I20240501 07:59:17.562575 5859 raft_consensus.cc:1261] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Refusing update from remote peer a3af86b0e9a04ecbb15d305c48d7acb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 175. Preceding OpId from leader: term: 2 index: 177. (index mismatch)
I20240501 07:59:17.562650 5728 raft_consensus.cc:1261] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a [term 2 FOLLOWER]: Refusing update from remote peer a3af86b0e9a04ecbb15d305c48d7acb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 175. Preceding OpId from leader: term: 2 index: 177. (index mismatch)
I20240501 07:59:17.563051 8206 consensus_queue.cc:1035] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 176, Last known committed idx: 175, Time since last communication: 0.000s
I20240501 07:59:17.563817 8206 consensus_queue.cc:1035] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 176, Last known committed idx: 175, Time since last communication: 0.000s
I20240501 07:59:17.566525 8207 mvcc.cc:205] Tried to move back new op lower bound from 7022798264570605568 to 7022798264026714112. Current Snapshot: MvccSnapshot[applied={T|T < 7022798264570605568}]
I20240501 07:59:17.705484 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:17.719583 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:17.721170 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:17.729415 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:17.745633 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:17.820328 5841 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:17.839198 5972 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:18.019591 7335 consensus_queue.cc:237] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 186, Committed index: 186, Last appended: 2.186, Last appended by leader: 175, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 187 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } }
I20240501 07:59:18.021833 5728 raft_consensus.cc:1261] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a [term 2 FOLLOWER]: Refusing update from remote peer a3af86b0e9a04ecbb15d305c48d7acb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 186. Preceding OpId from leader: term: 2 index: 187. (index mismatch)
I20240501 07:59:18.022956 8206 consensus_queue.cc:1035] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 187, Last known committed idx: 186, Time since last communication: 0.000s
I20240501 07:59:18.028103 8206 raft_consensus.cc:2932] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [term 2 LEADER]: Committing config change with OpId 2.187: config changed from index 71 to 187, VOTER f022e3632e9a428882a6cd16dc398723 (127.0.99.130) evicted. New config: { opid_index: 187 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } }
I20240501 07:59:18.029078 5728 raft_consensus.cc:2932] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a [term 2 FOLLOWER]: Committing config change with OpId 2.187: config changed from index 71 to 187, VOTER f022e3632e9a428882a6cd16dc398723 (127.0.99.130) evicted. New config: { opid_index: 187 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } }
I20240501 07:59:18.035629 5583 catalog_manager.cc:5484] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 reported cstate change: config changed from index 71 to 187, VOTER f022e3632e9a428882a6cd16dc398723 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" committed_config { opid_index: 187 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } }
I20240501 07:59:18.104174 5841 tablet_service.cc:1514] Processing DeleteTablet for tablet 877dfd1e30ee4e12b16674d96d57c48e with delete_type TABLET_DATA_TOMBSTONED (TS f022e3632e9a428882a6cd16dc398723 not found in new config with opid_index 187) from {username='slave'} at 127.0.0.1:54330
I20240501 07:59:18.113258 8304 tablet_replica.cc:330] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723: stopping tablet replica
I20240501 07:59:18.113575 8304 raft_consensus.cc:2229] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:18.113770 8304 raft_consensus.cc:2258] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:18.114681 8304 ts_tablet_manager.cc:1875] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:18.120553 8304 ts_tablet_manager.cc:1888] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.186
I20240501 07:59:18.123003 8304 log.cc:1196] T 877dfd1e30ee4e12b16674d96d57c48e P f022e3632e9a428882a6cd16dc398723: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/877dfd1e30ee4e12b16674d96d57c48e
I20240501 07:59:18.123934 5573 catalog_manager.cc:4830] TS f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187): tablet 877dfd1e30ee4e12b16674d96d57c48e (table rebalance_test_table_4 [id=2c9dba3a010e4d3fa551b89730285ff8]) successfully deleted
I20240501 07:59:18.403687 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:18.404148 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:18.414925 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:18.432771 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:18.433065 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:18.457140 5972 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:18.580251 5841 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:18.826543 5727 consensus_queue.cc:237] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 218, Committed index: 218, Last appended: 1.218, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:18.827934 7070 raft_consensus.cc:1261] T bd95a7eff75845c0b49790c9657ef1e0 P 72a601327d9d447aa7957cde6c36e000 [term 1 FOLLOWER]: Refusing update from remote peer 46c782f15fb643acb2ef72105f1fb04a: Log matching property violated. Preceding OpId in replica: term: 1 index: 218. Preceding OpId from leader: term: 1 index: 219. (index mismatch)
I20240501 07:59:18.828744 8161 consensus_queue.cc:1035] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20240501 07:59:18.832887 8161 raft_consensus.cc:2932] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a [term 1 LEADER]: Committing config change with OpId 1.219: config changed from index 50 to 219, VOTER f022e3632e9a428882a6cd16dc398723 (127.0.99.130) evicted. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:18.835860 5583 catalog_manager.cc:5484] T bd95a7eff75845c0b49790c9657ef1e0 P 46c782f15fb643acb2ef72105f1fb04a reported cstate change: config changed from index 50 to 219, VOTER f022e3632e9a428882a6cd16dc398723 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "46c782f15fb643acb2ef72105f1fb04a" committed_config { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:18.836578 7070 raft_consensus.cc:2932] T bd95a7eff75845c0b49790c9657ef1e0 P 72a601327d9d447aa7957cde6c36e000 [term 1 FOLLOWER]: Committing config change with OpId 1.219: config changed from index 50 to 219, VOTER f022e3632e9a428882a6cd16dc398723 (127.0.99.130) evicted. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:18.876382 5841 tablet_service.cc:1514] Processing DeleteTablet for tablet bd95a7eff75845c0b49790c9657ef1e0 with delete_type TABLET_DATA_TOMBSTONED (TS f022e3632e9a428882a6cd16dc398723 not found in new config with opid_index 219) from {username='slave'} at 127.0.0.1:54330
I20240501 07:59:18.882787 8372 tablet_replica.cc:330] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723: stopping tablet replica
I20240501 07:59:18.883004 8372 raft_consensus.cc:2229] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:18.883147 8372 raft_consensus.cc:2258] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:18.884749 8372 ts_tablet_manager.cc:1875] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:18.887498 8372 ts_tablet_manager.cc:1888] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.218
I20240501 07:59:18.887588 8372 log.cc:1196] T bd95a7eff75845c0b49790c9657ef1e0 P f022e3632e9a428882a6cd16dc398723: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/bd95a7eff75845c0b49790c9657ef1e0
I20240501 07:59:18.911887 5573 catalog_manager.cc:4830] TS f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187): tablet bd95a7eff75845c0b49790c9657ef1e0 (table rebalance_test_table_1 [id=b2c8da5392b24236b9cc0f5c750c0528]) successfully deleted
I20240501 07:59:19.103003 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:19.119907 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:19.144853 5841 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:19.175772 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:19.234097 5972 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:19.255208 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:19.266397 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:19.670718 5860 consensus_queue.cc:237] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 1.227, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } }
I20240501 07:59:19.671870 7335 raft_consensus.cc:1261] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 1 index: 227. Preceding OpId from leader: term: 1 index: 228. (index mismatch)
I20240501 07:59:19.672379 7473 consensus_queue.cc:1035] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s
I20240501 07:59:19.674680 8213 raft_consensus.cc:2932] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Committing config change with OpId 1.228: config changed from index 41 to 228, VOTER ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) evicted. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } }
I20240501 07:59:19.677630 5583 catalog_manager.cc:5484] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 reported cstate change: config changed from index 41 to 228, VOTER ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } }
I20240501 07:59:19.677597 7335 raft_consensus.cc:2932] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Committing config change with OpId 1.228: config changed from index 41 to 228, VOTER ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) evicted. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } }
I20240501 07:59:19.687965 5972 tablet_service.cc:1514] Processing DeleteTablet for tablet a07321bfc4c1445a906ec7c1261693e9 with delete_type TABLET_DATA_TOMBSTONED (TS ec372f1e72b040e780e6bdd4d5409d33 not found in new config with opid_index 228) from {username='slave'} at 127.0.0.1:38970
I20240501 07:59:19.688989 8435 tablet_replica.cc:330] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33: stopping tablet replica
I20240501 07:59:19.689184 8435 raft_consensus.cc:2229] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:19.689411 8435 raft_consensus.cc:2258] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:19.691525 8435 ts_tablet_manager.cc:1875] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:19.724979 8435 ts_tablet_manager.cc:1888] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.227
I20240501 07:59:19.725216 8435 log.cc:1196] T a07321bfc4c1445a906ec7c1261693e9 P ec372f1e72b040e780e6bdd4d5409d33: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a07321bfc4c1445a906ec7c1261693e9
I20240501 07:59:19.734779 5575 catalog_manager.cc:4830] TS ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531): tablet a07321bfc4c1445a906ec7c1261693e9 (table rebalance_test_table_1 [id=b2c8da5392b24236b9cc0f5c750c0528]) successfully deleted
I20240501 07:59:19.961928 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:20.001399 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:20.023691 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:20.062300 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:20.102880 5841 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:20.104300 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:20.134068 5972 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:20.271291 5991 consensus_queue.cc:237] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 1.237, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } }
I20240501 07:59:20.273758 6940 raft_consensus.cc:1261] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 1 index: 237. Preceding OpId from leader: term: 1 index: 238. (index mismatch)
I20240501 07:59:20.275625 8473 consensus_queue.cc:1035] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.001s
I20240501 07:59:20.289979 6611 raft_consensus.cc:2932] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Committing config change with OpId 1.238: config changed from index 40 to 238, VOTER 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129) evicted. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:20.290405 6937 raft_consensus.cc:2932] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Committing config change with OpId 1.238: config changed from index 40 to 238, VOTER 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129) evicted. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:20.295709 5583 catalog_manager.cc:5484] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 reported cstate change: config changed from index 40 to 238, VOTER 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:20.329298 5710 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f4b2166aae54490a36ced8d37caf7b1 with delete_type TABLET_DATA_TOMBSTONED (TS 46c782f15fb643acb2ef72105f1fb04a not found in new config with opid_index 238) from {username='slave'} at 127.0.0.1:48480
I20240501 07:59:20.329856 8502 tablet_replica.cc:330] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a: stopping tablet replica
I20240501 07:59:20.330204 8502 raft_consensus.cc:2229] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:20.330464 8502 raft_consensus.cc:2258] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:20.332074 8502 ts_tablet_manager.cc:1875] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:20.337641 8502 ts_tablet_manager.cc:1888] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.235
I20240501 07:59:20.337881 8502 log.cc:1196] T 7f4b2166aae54490a36ced8d37caf7b1 P 46c782f15fb643acb2ef72105f1fb04a: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7f4b2166aae54490a36ced8d37caf7b1
I20240501 07:59:20.341156 5575 catalog_manager.cc:4830] TS 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797): tablet 7f4b2166aae54490a36ced8d37caf7b1 (table rebalance_test_table_2 [id=f3b4504f12d4447ba80f01cbb31b8c0a]) successfully deleted
I20240501 07:59:20.605309 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:20.660645 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:20.709462 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:20.709945 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:20.715400 5972 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:20.750589 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:20.778227 5841 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:21.018795 5993 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "65e9dbbdb1fa459fabc258f9e6878f00"
dest_uuid: "ec372f1e72b040e780e6bdd4d5409d33"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34218
I20240501 07:59:21.018986 5993 raft_consensus.cc:595] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:21.035601 7534 raft_consensus.cc:979] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33: : Instructing follower f022e3632e9a428882a6cd16dc398723 to start an election
I20240501 07:59:21.035715 7534 raft_consensus.cc:1067] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Signalling peer f022e3632e9a428882a6cd16dc398723 to start an election
I20240501 07:59:21.040071 5862 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65e9dbbdb1fa459fabc258f9e6878f00"
dest_uuid: "f022e3632e9a428882a6cd16dc398723"
from {username='slave'} at 127.0.99.131:42307
I20240501 07:59:21.041427 5862 raft_consensus.cc:482] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:21.041532 5862 raft_consensus.cc:3037] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:21.042703 5862 raft_consensus.cc:504] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } }
I20240501 07:59:21.043972 5862 leader_election.cc:288] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 2 election: Requested vote from peers ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531), 12f3f54871994efeb833126452c61da9 (127.0.99.132:38041)
I20240501 07:59:21.044049 5991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65e9dbbdb1fa459fabc258f9e6878f00" candidate_uuid: "f022e3632e9a428882a6cd16dc398723" candidate_term: 2 candidate_status { last_received { term: 1 index: 235 } } ignore_live_leader: true dest_uuid: "ec372f1e72b040e780e6bdd4d5409d33"
I20240501 07:59:21.044179 5991 raft_consensus.cc:3032] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:21.044221 5991 raft_consensus.cc:729] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Running, Role: LEADER
I20240501 07:59:21.044376 5991 consensus_queue.cc:260] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 1.235, Last appended by leader: 235, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } }
I20240501 07:59:21.044540 5991 raft_consensus.cc:3037] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:21.045769 5991 raft_consensus.cc:2454] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f022e3632e9a428882a6cd16dc398723 in term 2.
I20240501 07:59:21.046339 5797 leader_election.cc:302] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec372f1e72b040e780e6bdd4d5409d33, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:21.048305 8213 raft_consensus.cc:2781] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:21.048394 8213 raft_consensus.cc:686] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [term 2 LEADER]: Becoming Leader. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Running, Role: LEADER
I20240501 07:59:21.048547 8213 consensus_queue.cc:237] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 1.235, Last appended by leader: 235, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } }
I20240501 07:59:21.052076 5580 catalog_manager.cc:5484] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 reported cstate change: term changed from 1 to 2, leader changed from ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) to f022e3632e9a428882a6cd16dc398723 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:21.077842 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65e9dbbdb1fa459fabc258f9e6878f00" candidate_uuid: "f022e3632e9a428882a6cd16dc398723" candidate_term: 2 candidate_status { last_received { term: 1 index: 235 } } ignore_live_leader: true dest_uuid: "12f3f54871994efeb833126452c61da9"
I20240501 07:59:21.078066 6937 raft_consensus.cc:3037] T 65e9dbbdb1fa459fabc258f9e6878f00 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:21.080039 6937 raft_consensus.cc:2454] T 65e9dbbdb1fa459fabc258f9e6878f00 P 12f3f54871994efeb833126452c61da9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f022e3632e9a428882a6cd16dc398723 in term 2.
W20240501 07:59:21.091280 5950 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38986: Illegal state: replica ec372f1e72b040e780e6bdd4d5409d33 is not leader of this config: current role FOLLOWER
I20240501 07:59:21.098969 5993 raft_consensus.cc:1261] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 2 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 1 index: 235. Preceding OpId from leader: term: 2 index: 237. (index mismatch)
I20240501 07:59:21.099424 7478 consensus_queue.cc:1035] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I20240501 07:59:21.104314 7832 mvcc.cc:205] Tried to move back new op lower bound from 7022798279056261120 to 7022798278856335360. Current Snapshot: MvccSnapshot[applied={T|T < 7022798279056261120}]
I20240501 07:59:21.107339 6937 raft_consensus.cc:1261] T 65e9dbbdb1fa459fabc258f9e6878f00 P 12f3f54871994efeb833126452c61da9 [term 2 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 1 index: 235. Preceding OpId from leader: term: 2 index: 237. (index mismatch)
I20240501 07:59:21.107810 6857 consensus_queue.cc:1035] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I20240501 07:59:21.387689 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:21.444741 5972 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:21.449262 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:21.455793 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:21.466228 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:21.497113 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:21.539608 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:21.850092 5860 consensus_queue.cc:237] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 261, Committed index: 261, Last appended: 2.261, Last appended by leader: 235, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } }
I20240501 07:59:21.851454 6939 raft_consensus.cc:1261] T 65e9dbbdb1fa459fabc258f9e6878f00 P 12f3f54871994efeb833126452c61da9 [term 2 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 2 index: 261. Preceding OpId from leader: term: 2 index: 262. (index mismatch)
I20240501 07:59:21.851933 6857 consensus_queue.cc:1035] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 262, Last known committed idx: 261, Time since last communication: 0.000s
I20240501 07:59:21.854727 8151 raft_consensus.cc:2932] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 [term 2 LEADER]: Committing config change with OpId 2.262: config changed from index 95 to 262, VOTER ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) evicted. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:21.858650 5580 catalog_manager.cc:5484] T 65e9dbbdb1fa459fabc258f9e6878f00 P f022e3632e9a428882a6cd16dc398723 reported cstate change: config changed from index 95 to 262, VOTER ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:21.861084 6939 raft_consensus.cc:2932] T 65e9dbbdb1fa459fabc258f9e6878f00 P 12f3f54871994efeb833126452c61da9 [term 2 FOLLOWER]: Committing config change with OpId 2.262: config changed from index 95 to 262, VOTER ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) evicted. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:21.920408 5972 tablet_service.cc:1514] Processing DeleteTablet for tablet 65e9dbbdb1fa459fabc258f9e6878f00 with delete_type TABLET_DATA_TOMBSTONED (TS ec372f1e72b040e780e6bdd4d5409d33 not found in new config with opid_index 262) from {username='slave'} at 127.0.0.1:38970
I20240501 07:59:21.936779 8642 tablet_replica.cc:330] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33: stopping tablet replica
I20240501 07:59:21.937600 8642 raft_consensus.cc:2229] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:21.937752 8642 raft_consensus.cc:2258] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:21.941138 8642 ts_tablet_manager.cc:1875] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:21.946326 8642 ts_tablet_manager.cc:1888] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.261
I20240501 07:59:21.946744 8642 log.cc:1196] T 65e9dbbdb1fa459fabc258f9e6878f00 P ec372f1e72b040e780e6bdd4d5409d33: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/65e9dbbdb1fa459fabc258f9e6878f00
I20240501 07:59:21.947980 5575 catalog_manager.cc:4830] TS ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531): tablet 65e9dbbdb1fa459fabc258f9e6878f00 (table rebalance_test_table_0 [id=3a5d5337264d45beb86597f2d8ffe305]) successfully deleted
I20240501 07:59:22.174561 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:22.196069 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:22.215881 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:22.219504 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:22.222043 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:22.275404 5972 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:22.288539 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:22.526821 5993 consensus_queue.cc:237] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 297, Committed index: 297, Last appended: 1.297, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:22.528880 7202 raft_consensus.cc:1261] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 1 index: 297. Preceding OpId from leader: term: 1 index: 298. (index mismatch)
I20240501 07:59:22.531230 8663 consensus_queue.cc:1035] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s
I20240501 07:59:22.537328 6611 raft_consensus.cc:2932] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Committing config change with OpId 1.298: config changed from index 86 to 298, VOTER 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129) evicted. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:22.540992 5583 catalog_manager.cc:5484] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 reported cstate change: config changed from index 86 to 298, VOTER 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:22.545090 7202 raft_consensus.cc:2932] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Committing config change with OpId 1.298: config changed from index 86 to 298, VOTER 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129) evicted. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:22.593698 5710 tablet_service.cc:1514] Processing DeleteTablet for tablet 0776eb886275492285da625d69e77e6b with delete_type TABLET_DATA_TOMBSTONED (TS 46c782f15fb643acb2ef72105f1fb04a not found in new config with opid_index 298) from {username='slave'} at 127.0.0.1:48480
I20240501 07:59:22.617502 8712 tablet_replica.cc:330] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a: stopping tablet replica
I20240501 07:59:22.617719 8712 raft_consensus.cc:2229] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:22.617935 8712 raft_consensus.cc:2258] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:22.619026 8712 ts_tablet_manager.cc:1875] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:22.622102 8712 ts_tablet_manager.cc:1888] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.297
I20240501 07:59:22.622285 8712 log.cc:1196] T 0776eb886275492285da625d69e77e6b P 46c782f15fb643acb2ef72105f1fb04a: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0776eb886275492285da625d69e77e6b
I20240501 07:59:22.623515 5575 catalog_manager.cc:4830] TS 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797): tablet 0776eb886275492285da625d69e77e6b (table rebalance_test_table_3 [id=62d90ece49d2427fbd192ff552b40579]) successfully deleted
I20240501 07:59:22.839371 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:22.844871 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:22.846622 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:22.880496 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:22.881398 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:22.925395 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:22.938148 5972 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:23.253017 5862 consensus_queue.cc:237] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 1.285, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } }
I20240501 07:59:23.256129 7335 raft_consensus.cc:1261] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 1 index: 285. Preceding OpId from leader: term: 1 index: 286. (index mismatch)
I20240501 07:59:23.260437 6857 consensus_queue.cc:1035] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 284, Time since last communication: 0.000s
I20240501 07:59:23.264925 7473 raft_consensus.cc:2932] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Committing config change with OpId 1.286: config changed from index 68 to 286, VOTER 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129) evicted. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } }
I20240501 07:59:23.266014 7335 raft_consensus.cc:2932] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Committing config change with OpId 1.286: config changed from index 68 to 286, VOTER 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129) evicted. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } }
I20240501 07:59:23.269497 5580 catalog_manager.cc:5484] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4 reported cstate change: config changed from index 68 to 286, VOTER 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } }
I20240501 07:59:23.309273 5710 tablet_service.cc:1514] Processing DeleteTablet for tablet 114e9265cef4425799ad44e73e053b59 with delete_type TABLET_DATA_TOMBSTONED (TS 46c782f15fb643acb2ef72105f1fb04a not found in new config with opid_index 286) from {username='slave'} at 127.0.0.1:48480
I20240501 07:59:23.318325 8787 tablet_replica.cc:330] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a: stopping tablet replica
I20240501 07:59:23.318604 8787 raft_consensus.cc:2229] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:23.318750 8787 pending_rounds.cc:62] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a: Trying to abort 1 pending ops.
I20240501 07:59:23.318871 8787 pending_rounds.cc:68] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a: Aborting op as it isn't in flight: id { term: 1 index: 285 } timestamp: 7022798287880450048 op_type: WRITE_OP write_request { tablet_id: "114e9265cef4425799ad44e73e053b59" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\241Dlw%~\304\362\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""70cae3fab07ad7e0" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022798287722622976 authz_token { token_data: "\010\373\356\307\261\006\"3\n\005slave\022*\n 3a5d5337264d45beb86597f2d8ffe305\020\001\030\001 \001(\001" signature: "<redacted>""*\321=\243U\226\336\033\"\341$\n\267\020\222Q\013\206G\324\371\260!\301:\371\030\200)V\325\020\374\203M4l\314\360\035\034\363&0H\300N\206\\\350\366\017\333B\372\215L\347\333a*R\374\243" signing_key_seq_num: 0 } } request_id { client_id: "88a20f6cb3074d7b820d65277db46444" seq_no: 857 first_incomplete_seq_no: 857 attempt_no: 0 }
I20240501 07:59:23.319422 8787 raft_consensus.cc:2258] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:23.321311 8787 ts_tablet_manager.cc:1875] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:23.329625 8787 ts_tablet_manager.cc:1888] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.285
I20240501 07:59:23.329815 8787 log.cc:1196] T 114e9265cef4425799ad44e73e053b59 P 46c782f15fb643acb2ef72105f1fb04a: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/114e9265cef4425799ad44e73e053b59
I20240501 07:59:23.330922 5575 catalog_manager.cc:4830] TS 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797): tablet 114e9265cef4425799ad44e73e053b59 (table rebalance_test_table_0 [id=3a5d5337264d45beb86597f2d8ffe305]) successfully deleted
I20240501 07:59:23.560751 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:23.562340 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:23.563288 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:23.585337 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:23.616714 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:23.629725 5972 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:23.640561 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:23.871237 5993 consensus_queue.cc:237] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 1.321, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:23.874265 7201 raft_consensus.cc:1261] T 3f187bcb4c494805a8d6f366732b695a P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 1 index: 321. Preceding OpId from leader: term: 1 index: 322. (index mismatch)
I20240501 07:59:23.874763 6611 consensus_queue.cc:1035] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.000s
I20240501 07:59:23.876528 7534 raft_consensus.cc:2932] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Committing config change with OpId 1.322: config changed from index 55 to 322, VOTER 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129) evicted. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:23.879240 7201 raft_consensus.cc:2932] T 3f187bcb4c494805a8d6f366732b695a P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Committing config change with OpId 1.322: config changed from index 55 to 322, VOTER 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129) evicted. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:23.882261 5582 catalog_manager.cc:5484] T 3f187bcb4c494805a8d6f366732b695a P ec372f1e72b040e780e6bdd4d5409d33 reported cstate change: config changed from index 55 to 322, VOTER 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:23.888301 5710 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f187bcb4c494805a8d6f366732b695a with delete_type TABLET_DATA_TOMBSTONED (TS 46c782f15fb643acb2ef72105f1fb04a not found in new config with opid_index 322) from {username='slave'} at 127.0.0.1:48480
I20240501 07:59:23.923894 8848 tablet_replica.cc:330] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a: stopping tablet replica
I20240501 07:59:23.925772 8848 raft_consensus.cc:2229] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:23.930565 8848 raft_consensus.cc:2258] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:23.933670 8848 ts_tablet_manager.cc:1875] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:23.936410 8848 ts_tablet_manager.cc:1888] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.321
I20240501 07:59:23.939218 8848 log.cc:1196] T 3f187bcb4c494805a8d6f366732b695a P 46c782f15fb643acb2ef72105f1fb04a: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/3f187bcb4c494805a8d6f366732b695a
I20240501 07:59:23.940016 5575 catalog_manager.cc:4830] TS 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797): tablet 3f187bcb4c494805a8d6f366732b695a (table rebalance_test_table_1 [id=b2c8da5392b24236b9cc0f5c750c0528]) successfully deleted
I20240501 07:59:24.112625 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:24.130791 5972 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:24.135721 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:24.151448 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:24.192878 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:24.200263 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:24.207774 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:24.328636 5993 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a6cb1009e4fb4fb0a8e7d55c2c24ff65"
dest_uuid: "ec372f1e72b040e780e6bdd4d5409d33"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34330
I20240501 07:59:24.328868 5993 raft_consensus.cc:595] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Received request to transfer leadership
W20240501 07:59:24.342597 5952 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.348389 5952 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.358644 5950 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.378347 5950 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.386873 5950 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.400359 5950 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.417147 5950 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.432194 5950 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.444590 5950 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.457038 5950 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.468417 5950 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.486222 5950 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.503737 5950 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.518252 5950 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.538754 5950 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.559052 5950 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
I20240501 07:59:24.560359 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:24.578140 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:24.580790 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:24.585062 5972 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 07:59:24.585774 5950 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
I20240501 07:59:24.586881 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 07:59:24.607270 5952 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
I20240501 07:59:24.613145 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:24.614511 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 07:59:24.629462 5950 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.654150 5952 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.676000 5952 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.700289 5950 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.724727 5952 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.754340 5952 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.781383 5952 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.810300 5952 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
W20240501 07:59:24.839043 5952 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Service unavailable: leader transfer in progress
I20240501 07:59:24.839764 8663 raft_consensus.cc:979] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33: : Instructing follower f022e3632e9a428882a6cd16dc398723 to start an election
I20240501 07:59:24.839848 8663 raft_consensus.cc:1067] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Signalling peer f022e3632e9a428882a6cd16dc398723 to start an election
I20240501 07:59:24.841411 5862 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a6cb1009e4fb4fb0a8e7d55c2c24ff65"
dest_uuid: "f022e3632e9a428882a6cd16dc398723"
from {username='slave'} at 127.0.99.131:42307
I20240501 07:59:24.841555 5862 raft_consensus.cc:482] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:24.841611 5862 raft_consensus.cc:3037] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:24.843932 5862 raft_consensus.cc:504] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:24.844404 5862 leader_election.cc:288] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 2 election: Requested vote from peers ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531), 72a601327d9d447aa7957cde6c36e000 (127.0.99.133:34267)
I20240501 07:59:24.845461 5991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6cb1009e4fb4fb0a8e7d55c2c24ff65" candidate_uuid: "f022e3632e9a428882a6cd16dc398723" candidate_term: 2 candidate_status { last_received { term: 1 index: 372 } } ignore_live_leader: true dest_uuid: "ec372f1e72b040e780e6bdd4d5409d33"
I20240501 07:59:24.845602 5991 raft_consensus.cc:3032] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:24.845646 5991 raft_consensus.cc:729] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Running, Role: LEADER
I20240501 07:59:24.845782 5991 consensus_queue.cc:260] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 372, Committed index: 372, Last appended: 1.372, Last appended by leader: 372, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:24.845929 5991 raft_consensus.cc:3037] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:24.849408 5991 raft_consensus.cc:2454] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f022e3632e9a428882a6cd16dc398723 in term 2.
I20240501 07:59:24.850047 7070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6cb1009e4fb4fb0a8e7d55c2c24ff65" candidate_uuid: "f022e3632e9a428882a6cd16dc398723" candidate_term: 2 candidate_status { last_received { term: 1 index: 372 } } ignore_live_leader: true dest_uuid: "72a601327d9d447aa7957cde6c36e000"
I20240501 07:59:24.850270 7070 raft_consensus.cc:3037] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P 72a601327d9d447aa7957cde6c36e000 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:24.851599 7070 raft_consensus.cc:2454] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P 72a601327d9d447aa7957cde6c36e000 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f022e3632e9a428882a6cd16dc398723 in term 2.
I20240501 07:59:24.851778 5797 leader_election.cc:302] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec372f1e72b040e780e6bdd4d5409d33, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:24.851985 8690 raft_consensus.cc:2781] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:24.852085 8690 raft_consensus.cc:686] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [term 2 LEADER]: Becoming Leader. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Running, Role: LEADER
I20240501 07:59:24.852228 8690 consensus_queue.cc:237] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 372, Committed index: 372, Last appended: 1.372, Last appended by leader: 372, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:24.853979 5580 catalog_manager.cc:5484] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 reported cstate change: term changed from 1 to 2, leader changed from ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) to f022e3632e9a428882a6cd16dc398723 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
W20240501 07:59:24.870319 5952 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39014: Illegal state: replica ec372f1e72b040e780e6bdd4d5409d33 is not leader of this config: current role FOLLOWER
I20240501 07:59:24.972530 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:24.976670 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:24.977169 7070 raft_consensus.cc:1261] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P 72a601327d9d447aa7957cde6c36e000 [term 2 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 1 index: 372. Preceding OpId from leader: term: 2 index: 374. (index mismatch)
I20240501 07:59:24.978242 8213 consensus_queue.cc:1035] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 373, Last known committed idx: 372, Time since last communication: 0.000s
I20240501 07:59:24.980005 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:24.989318 5991 raft_consensus.cc:1261] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 2 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 1 index: 372. Preceding OpId from leader: term: 2 index: 374. (index mismatch)
I20240501 07:59:24.990358 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:24.995240 8690 consensus_queue.cc:1035] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 373, Last known committed idx: 372, Time since last communication: 0.000s
I20240501 07:59:25.042377 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:25.062780 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:25.071331 5841 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:25.245424 5725 consensus_queue.cc:237] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 368, Committed index: 368, Last appended: 1.368, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } }
I20240501 07:59:25.246773 6939 raft_consensus.cc:1261] T dad26af7f84e42208fd4461656117035 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Refusing update from remote peer 46c782f15fb643acb2ef72105f1fb04a: Log matching property violated. Preceding OpId in replica: term: 1 index: 368. Preceding OpId from leader: term: 1 index: 369. (index mismatch)
I20240501 07:59:25.247509 8952 consensus_queue.cc:1035] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 369, Last known committed idx: 368, Time since last communication: 0.000s
I20240501 07:59:25.249814 8892 raft_consensus.cc:2932] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a [term 1 LEADER]: Committing config change with OpId 1.369: config changed from index 71 to 369, VOTER f022e3632e9a428882a6cd16dc398723 (127.0.99.130) evicted. New config: { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:25.251787 6939 raft_consensus.cc:2932] T dad26af7f84e42208fd4461656117035 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Committing config change with OpId 1.369: config changed from index 71 to 369, VOTER f022e3632e9a428882a6cd16dc398723 (127.0.99.130) evicted. New config: { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:25.254314 5588 catalog_manager.cc:5484] T dad26af7f84e42208fd4461656117035 P 46c782f15fb643acb2ef72105f1fb04a reported cstate change: config changed from index 71 to 369, VOTER f022e3632e9a428882a6cd16dc398723 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "46c782f15fb643acb2ef72105f1fb04a" committed_config { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:25.257961 5841 tablet_service.cc:1514] Processing DeleteTablet for tablet dad26af7f84e42208fd4461656117035 with delete_type TABLET_DATA_TOMBSTONED (TS f022e3632e9a428882a6cd16dc398723 not found in new config with opid_index 369) from {username='slave'} at 127.0.0.1:54330
I20240501 07:59:25.259232 9000 tablet_replica.cc:330] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723: stopping tablet replica
I20240501 07:59:25.259624 9000 raft_consensus.cc:2229] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:25.259847 9000 raft_consensus.cc:2258] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:25.261677 9000 ts_tablet_manager.cc:1875] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:25.265571 9000 ts_tablet_manager.cc:1888] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.368
I20240501 07:59:25.265728 9000 log.cc:1196] T dad26af7f84e42208fd4461656117035 P f022e3632e9a428882a6cd16dc398723: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/dad26af7f84e42208fd4461656117035
I20240501 07:59:25.266381 5573 catalog_manager.cc:4830] TS f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187): tablet dad26af7f84e42208fd4461656117035 (table rebalance_test_table_3 [id=62d90ece49d2427fbd192ff552b40579]) successfully deleted
I20240501 07:59:25.526633 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:25.555464 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:25.555584 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:25.566845 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:25.571378 5841 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:25.583917 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:25.584496 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:25.872040 5993 consensus_queue.cc:237] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 437, Committed index: 437, Last appended: 1.437, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 438 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } }
I20240501 07:59:25.898046 6938 raft_consensus.cc:1261] T 2a577537e9294fd5bd8507232d1d1a76 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 1 index: 437. Preceding OpId from leader: term: 1 index: 438. (index mismatch)
I20240501 07:59:25.899145 8999 consensus_queue.cc:1035] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 438, Last known committed idx: 437, Time since last communication: 0.000s
I20240501 07:59:25.909746 6611 raft_consensus.cc:2932] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Committing config change with OpId 1.438: config changed from index 64 to 438, VOTER 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129) evicted. New config: { opid_index: 438 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:25.910995 6938 raft_consensus.cc:2932] T 2a577537e9294fd5bd8507232d1d1a76 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Committing config change with OpId 1.438: config changed from index 64 to 438, VOTER 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129) evicted. New config: { opid_index: 438 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:25.915030 5588 catalog_manager.cc:5484] T 2a577537e9294fd5bd8507232d1d1a76 P ec372f1e72b040e780e6bdd4d5409d33 reported cstate change: config changed from index 64 to 438, VOTER 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: 438 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } }
I20240501 07:59:25.961184 5710 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a577537e9294fd5bd8507232d1d1a76 with delete_type TABLET_DATA_TOMBSTONED (TS 46c782f15fb643acb2ef72105f1fb04a not found in new config with opid_index 438) from {username='slave'} at 127.0.0.1:48480
I20240501 07:59:25.967615 9074 tablet_replica.cc:330] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a: stopping tablet replica
I20240501 07:59:25.980469 9074 raft_consensus.cc:2229] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:25.980674 9074 raft_consensus.cc:2258] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:25.982033 9074 ts_tablet_manager.cc:1875] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:25.984946 9074 ts_tablet_manager.cc:1888] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.437
I20240501 07:59:25.985872 9074 log.cc:1196] T 2a577537e9294fd5bd8507232d1d1a76 P 46c782f15fb643acb2ef72105f1fb04a: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/2a577537e9294fd5bd8507232d1d1a76
I20240501 07:59:25.988113 5575 catalog_manager.cc:4830] TS 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797): tablet 2a577537e9294fd5bd8507232d1d1a76 (table rebalance_test_table_4 [id=2c9dba3a010e4d3fa551b89730285ff8]) successfully deleted
I20240501 07:59:26.256966 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:26.257690 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:26.272303 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:26.284705 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:26.301383 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:26.301395 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:26.340318 5841 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:26.505865 5862 consensus_queue.cc:237] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 484, Committed index: 484, Last appended: 1.484, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 485 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:26.512214 7068 raft_consensus.cc:1261] T 26ad4c1ac47544f586d3f46f5dc9c77e P 72a601327d9d447aa7957cde6c36e000 [term 1 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 1 index: 484. Preceding OpId from leader: term: 1 index: 485. (index mismatch)
I20240501 07:59:26.518872 8213 consensus_queue.cc:1035] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 485, Last known committed idx: 484, Time since last communication: 0.000s
I20240501 07:59:26.526935 8690 raft_consensus.cc:2932] T 26ad4c1ac47544f586d3f46f5dc9c77e P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Committing config change with OpId 1.485: config changed from index 67 to 485, VOTER ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) evicted. New config: { opid_index: 485 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:26.528949 7070 raft_consensus.cc:2932] T 26ad4c1ac47544f586d3f46f5dc9c77e P 72a601327d9d447aa7957cde6c36e000 [term 1 FOLLOWER]: Committing config change with OpId 1.485: config changed from index 67 to 485, VOTER ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) evicted. New config: { opid_index: 485 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:26.531862 5583 catalog_manager.cc:5484] T 26ad4c1ac47544f586d3f46f5dc9c77e P 72a601327d9d447aa7957cde6c36e000 reported cstate change: config changed from index 67 to 485, VOTER ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: 485 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:26.542053 5972 tablet_service.cc:1514] Processing DeleteTablet for tablet 26ad4c1ac47544f586d3f46f5dc9c77e with delete_type TABLET_DATA_TOMBSTONED (TS ec372f1e72b040e780e6bdd4d5409d33 not found in new config with opid_index 485) from {username='slave'} at 127.0.0.1:38970
I20240501 07:59:26.542886 9139 tablet_replica.cc:330] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33: stopping tablet replica
I20240501 07:59:26.543603 9139 raft_consensus.cc:2229] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:26.543807 9139 raft_consensus.cc:2258] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:26.544498 9139 ts_tablet_manager.cc:1875] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:26.547191 9139 ts_tablet_manager.cc:1888] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.484
I20240501 07:59:26.547349 9139 log.cc:1196] T 26ad4c1ac47544f586d3f46f5dc9c77e P ec372f1e72b040e780e6bdd4d5409d33: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/26ad4c1ac47544f586d3f46f5dc9c77e
I20240501 07:59:26.548460 5575 catalog_manager.cc:4830] TS ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531): tablet 26ad4c1ac47544f586d3f46f5dc9c77e (table rebalance_test_table_4 [id=2c9dba3a010e4d3fa551b89730285ff8]) successfully deleted
I20240501 07:59:27.377127 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:27.388962 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:27.395964 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:27.412158 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:27.427026 5841 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:27.427151 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:27.428694 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:27.605152 5862 consensus_queue.cc:237] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 462, Committed index: 462, Last appended: 2.462, Last appended by leader: 372, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 463 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } }
I20240501 07:59:27.608494 7068 raft_consensus.cc:1261] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P 72a601327d9d447aa7957cde6c36e000 [term 2 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 2 index: 462. Preceding OpId from leader: term: 2 index: 463. (index mismatch)
I20240501 07:59:27.609309 8690 consensus_queue.cc:1035] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 463, Last known committed idx: 462, Time since last communication: 0.000s
I20240501 07:59:27.614941 8770 raft_consensus.cc:2932] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 [term 2 LEADER]: Committing config change with OpId 2.463: config changed from index 91 to 463, VOTER ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) evicted. New config: { opid_index: 463 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:27.617326 7070 raft_consensus.cc:2932] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P 72a601327d9d447aa7957cde6c36e000 [term 2 FOLLOWER]: Committing config change with OpId 2.463: config changed from index 91 to 463, VOTER ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) evicted. New config: { opid_index: 463 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:27.618139 5589 catalog_manager.cc:5484] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P f022e3632e9a428882a6cd16dc398723 reported cstate change: config changed from index 91 to 463, VOTER ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: 463 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "72a601327d9d447aa7957cde6c36e000" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34267 } } }
I20240501 07:59:27.624504 5972 tablet_service.cc:1514] Processing DeleteTablet for tablet a6cb1009e4fb4fb0a8e7d55c2c24ff65 with delete_type TABLET_DATA_TOMBSTONED (TS ec372f1e72b040e780e6bdd4d5409d33 not found in new config with opid_index 463) from {username='slave'} at 127.0.0.1:38970
I20240501 07:59:27.624900 9264 tablet_replica.cc:330] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33: stopping tablet replica
I20240501 07:59:27.625145 9264 raft_consensus.cc:2229] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:27.625303 9264 raft_consensus.cc:2258] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:27.626039 9264 ts_tablet_manager.cc:1875] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:27.628732 9264 ts_tablet_manager.cc:1888] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.462
I20240501 07:59:27.628916 9264 log.cc:1196] T a6cb1009e4fb4fb0a8e7d55c2c24ff65 P ec372f1e72b040e780e6bdd4d5409d33: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a6cb1009e4fb4fb0a8e7d55c2c24ff65
I20240501 07:59:27.630707 5575 catalog_manager.cc:4830] TS ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531): tablet a6cb1009e4fb4fb0a8e7d55c2c24ff65 (table rebalance_test_table_3 [id=62d90ece49d2427fbd192ff552b40579]) successfully deleted
I20240501 07:59:27.917879 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:27.919237 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:27.919288 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:27.950767 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:27.971163 5841 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:27.985486 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:28.008808 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:28.168082 5862 consensus_queue.cc:237] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 447, Committed index: 447, Last appended: 1.447, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
W20240501 07:59:28.170327 5795 consensus_peers.cc:485] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 -> Peer afc8c07c29354a04926027f57b8c6fab (127.0.99.134:38017): Couldn't send request to peer afc8c07c29354a04926027f57b8c6fab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 114e9265cef4425799ad44e73e053b59. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:28.181645 7335 raft_consensus.cc:1261] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 1 index: 447. Preceding OpId from leader: term: 1 index: 448. (index mismatch)
I20240501 07:59:28.183423 8770 consensus_queue.cc:1035] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 448, Last known committed idx: 446, Time since last communication: 0.000s
I20240501 07:59:28.196959 8690 raft_consensus.cc:2932] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Committing config change with OpId 1.448: config changed from index 286 to 448, VOTER afc8c07c29354a04926027f57b8c6fab (127.0.99.134) added. New config: { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:28.200222 5583 catalog_manager.cc:5484] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 reported cstate change: config changed from index 286 to 448, VOTER afc8c07c29354a04926027f57b8c6fab (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:28.213912 7337 raft_consensus.cc:2932] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Committing config change with OpId 1.448: config changed from index 286 to 448, VOTER afc8c07c29354a04926027f57b8c6fab (127.0.99.134) added. New config: { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:28.436092 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:28.463702 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:28.478196 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:28.484571 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:28.506256 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:28.524411 5841 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:28.564018 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:28.580657 9394 ts_tablet_manager.cc:902] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab: Initiating tablet copy from peer f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187)
I20240501 07:59:28.581324 9394 tablet_copy_client.cc:308] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42187
I20240501 07:59:28.584412 5882 tablet_copy_service.cc:140] P f022e3632e9a428882a6cd16dc398723: Received BeginTabletCopySession request for tablet 114e9265cef4425799ad44e73e053b59 from peer afc8c07c29354a04926027f57b8c6fab ({username='slave'} at 127.0.99.134:41081)
I20240501 07:59:28.584547 5882 tablet_copy_service.cc:161] P f022e3632e9a428882a6cd16dc398723: Beginning new tablet copy session on tablet 114e9265cef4425799ad44e73e053b59 from peer afc8c07c29354a04926027f57b8c6fab at {username='slave'} at 127.0.99.134:41081: session id = afc8c07c29354a04926027f57b8c6fab-114e9265cef4425799ad44e73e053b59
I20240501 07:59:28.585542 5882 tablet_copy_source_session.cc:205] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:28.587308 9394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 114e9265cef4425799ad44e73e053b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:28.589738 9394 tablet_copy_client.cc:783] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:28.589898 9394 tablet_copy_client.cc:647] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:28.595230 9394 tablet_copy_client.cc:523] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:28.597793 9394 tablet_bootstrap.cc:492] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab: Bootstrap starting.
I20240501 07:59:28.999716 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:29.027732 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:29.030325 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:29.073410 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:29.075794 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:29.075904 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:29.081724 5841 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:29.336479 9394 tablet_bootstrap.cc:492] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab: Bootstrap replayed 1/1 log segments. Stats: ops{read=466 overwritten=0 applied=466 ignored=0} inserts{seen=1149 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:29.336953 9394 tablet_bootstrap.cc:492] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab: Bootstrap complete.
I20240501 07:59:29.337344 9394 ts_tablet_manager.cc:1367] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab: Time spent bootstrapping tablet: real 0.740s user 0.344s sys 0.021s
I20240501 07:59:29.337940 9394 raft_consensus.cc:348] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:29.338294 9394 raft_consensus.cc:729] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc8c07c29354a04926027f57b8c6fab, State: Initialized, Role: FOLLOWER
I20240501 07:59:29.338475 9394 consensus_queue.cc:260] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 466, Last appended: 1.466, Last appended by leader: 466, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:29.338981 9394 ts_tablet_manager.cc:1398] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:29.339823 5882 tablet_copy_service.cc:342] P f022e3632e9a428882a6cd16dc398723: Request end of tablet copy session afc8c07c29354a04926027f57b8c6fab-114e9265cef4425799ad44e73e053b59 received from {username='slave'} at 127.0.99.134:41081
I20240501 07:59:29.339903 5882 tablet_copy_service.cc:433] P f022e3632e9a428882a6cd16dc398723: ending tablet copy session afc8c07c29354a04926027f57b8c6fab-114e9265cef4425799ad44e73e053b59 on tablet 114e9265cef4425799ad44e73e053b59 with peer afc8c07c29354a04926027f57b8c6fab
I20240501 07:59:29.592350 7203 raft_consensus.cc:1203] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.447->[1.448-1.497] Dedup: 1.466->[1.467-1.497]
I20240501 07:59:29.616845 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:29.626632 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:29.638018 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:29.651274 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:29.658269 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:29.675412 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:29.731405 5841 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:29.796162 5862 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "114e9265cef4425799ad44e73e053b59"
dest_uuid: "f022e3632e9a428882a6cd16dc398723"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60680
I20240501 07:59:29.796319 5862 raft_consensus.cc:595] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Received request to transfer leadership
W20240501 07:59:29.857645 5821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
W20240501 07:59:29.861693 5821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
W20240501 07:59:29.868593 5821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
W20240501 07:59:29.880091 5821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
W20240501 07:59:29.891456 5821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
W20240501 07:59:29.901198 5821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
W20240501 07:59:29.912071 5821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
W20240501 07:59:29.924016 5821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
W20240501 07:59:29.937155 5821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
W20240501 07:59:29.951406 5821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
W20240501 07:59:29.965054 5821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
I20240501 07:59:29.973742 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:29.973769 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 07:59:29.979707 5821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
I20240501 07:59:29.988459 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 07:59:29.994323 5822 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
I20240501 07:59:29.997102 7170 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:29.999248 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 07:59:30.012617 5821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
I20240501 07:59:30.012660 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:30.013015 5841 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 07:59:30.030230 5821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
W20240501 07:59:30.049571 5821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
W20240501 07:59:30.071520 5821 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
W20240501 07:59:30.096372 5822 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
W20240501 07:59:30.117487 5822 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
I20240501 07:59:30.142248 8690 raft_consensus.cc:979] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723: : Instructing follower afc8c07c29354a04926027f57b8c6fab to start an election
I20240501 07:59:30.142375 8690 raft_consensus.cc:1067] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Signalling peer afc8c07c29354a04926027f57b8c6fab to start an election
I20240501 07:59:30.143529 7200 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "114e9265cef4425799ad44e73e053b59"
dest_uuid: "afc8c07c29354a04926027f57b8c6fab"
from {username='slave'} at 127.0.99.130:54153
I20240501 07:59:30.143752 7200 raft_consensus.cc:482] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:30.143831 7200 raft_consensus.cc:3037] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:30.145017 7200 raft_consensus.cc:504] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:30.145833 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114e9265cef4425799ad44e73e053b59" candidate_uuid: "afc8c07c29354a04926027f57b8c6fab" candidate_term: 2 candidate_status { last_received { term: 1 index: 500 } } ignore_live_leader: true dest_uuid: "f022e3632e9a428882a6cd16dc398723"
I20240501 07:59:30.145957 5862 raft_consensus.cc:3032] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:30.146001 5862 raft_consensus.cc:729] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Running, Role: LEADER
W20240501 07:59:30.146004 5822 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Service unavailable: leader transfer in progress
I20240501 07:59:30.146167 5862 consensus_queue.cc:260] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 500, Committed index: 500, Last appended: 1.500, Last appended by leader: 500, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:30.146318 5862 raft_consensus.cc:3037] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:30.146500 7200 leader_election.cc:288] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab [CANDIDATE]: Term 2 election: Requested vote from peers f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187), a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135:45001)
I20240501 07:59:30.146977 7337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114e9265cef4425799ad44e73e053b59" candidate_uuid: "afc8c07c29354a04926027f57b8c6fab" candidate_term: 2 candidate_status { last_received { term: 1 index: 500 } } ignore_live_leader: true dest_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4"
I20240501 07:59:30.147127 7337 raft_consensus.cc:3037] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:30.147614 5862 raft_consensus.cc:2454] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afc8c07c29354a04926027f57b8c6fab in term 2.
I20240501 07:59:30.148172 7337 raft_consensus.cc:2454] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afc8c07c29354a04926027f57b8c6fab in term 2.
I20240501 07:59:30.148578 7135 leader_election.cc:302] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3af86b0e9a04ecbb15d305c48d7acb4, afc8c07c29354a04926027f57b8c6fab; no voters:
I20240501 07:59:30.148900 9540 raft_consensus.cc:2781] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:30.149170 9540 raft_consensus.cc:686] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab [term 2 LEADER]: Becoming Leader. State: Replica: afc8c07c29354a04926027f57b8c6fab, State: Running, Role: LEADER
I20240501 07:59:30.149590 9540 consensus_queue.cc:237] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 500, Committed index: 500, Last appended: 1.500, Last appended by leader: 500, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:30.152102 5586 catalog_manager.cc:5484] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab reported cstate change: term changed from 1 to 2, leader changed from f022e3632e9a428882a6cd16dc398723 (127.0.99.130) to afc8c07c29354a04926027f57b8c6fab (127.0.99.134). New cstate: current_term: 2 leader_uuid: "afc8c07c29354a04926027f57b8c6fab" committed_config { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
W20240501 07:59:30.171320 5822 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59672: Illegal state: replica f022e3632e9a428882a6cd16dc398723 is not leader of this config: current role FOLLOWER
I20240501 07:59:30.251530 7337 raft_consensus.cc:1261] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 2 FOLLOWER]: Refusing update from remote peer afc8c07c29354a04926027f57b8c6fab: Log matching property violated. Preceding OpId in replica: term: 1 index: 500. Preceding OpId from leader: term: 2 index: 502. (index mismatch)
I20240501 07:59:30.251855 5862 raft_consensus.cc:1261] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Refusing update from remote peer afc8c07c29354a04926027f57b8c6fab: Log matching property violated. Preceding OpId in replica: term: 1 index: 500. Preceding OpId from leader: term: 2 index: 502. (index mismatch)
I20240501 07:59:30.251956 9541 consensus_queue.cc:1035] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 501, Last known committed idx: 500, Time since last communication: 0.000s
I20240501 07:59:30.253968 9540 consensus_queue.cc:1035] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 501, Last known committed idx: 500, Time since last communication: 0.000s
I20240501 07:59:30.257037 9472 mvcc.cc:205] Tried to move back new op lower bound from 7022798316544663552 to 7022798316136538112. Current Snapshot: MvccSnapshot[applied={T|T < 7022798316544663552}]
I20240501 07:59:30.537247 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:30.541383 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:30.547156 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:30.565861 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:30.577961 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:30.579998 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:30.620388 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:30.773860 7200 consensus_queue.cc:237] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 515, Committed index: 515, Last appended: 2.515, Last appended by leader: 500, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 516 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } }
I20240501 07:59:30.775358 7335 raft_consensus.cc:1261] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 2 FOLLOWER]: Refusing update from remote peer afc8c07c29354a04926027f57b8c6fab: Log matching property violated. Preceding OpId in replica: term: 2 index: 515. Preceding OpId from leader: term: 2 index: 516. (index mismatch)
I20240501 07:59:30.775838 9582 consensus_queue.cc:1035] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 516, Last known committed idx: 515, Time since last communication: 0.000s
I20240501 07:59:30.778673 9548 raft_consensus.cc:2932] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab [term 2 LEADER]: Committing config change with OpId 2.516: config changed from index 448 to 516, VOTER f022e3632e9a428882a6cd16dc398723 (127.0.99.130) evicted. New config: { opid_index: 516 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:30.779080 7335 raft_consensus.cc:2932] T 114e9265cef4425799ad44e73e053b59 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 2 FOLLOWER]: Committing config change with OpId 2.516: config changed from index 448 to 516, VOTER f022e3632e9a428882a6cd16dc398723 (127.0.99.130) evicted. New config: { opid_index: 516 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:30.782074 5582 catalog_manager.cc:5484] T 114e9265cef4425799ad44e73e053b59 P afc8c07c29354a04926027f57b8c6fab reported cstate change: config changed from index 448 to 516, VOTER f022e3632e9a428882a6cd16dc398723 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "afc8c07c29354a04926027f57b8c6fab" committed_config { opid_index: 516 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } }
I20240501 07:59:30.786327 5841 tablet_service.cc:1514] Processing DeleteTablet for tablet 114e9265cef4425799ad44e73e053b59 with delete_type TABLET_DATA_TOMBSTONED (TS f022e3632e9a428882a6cd16dc398723 not found in new config with opid_index 516) from {username='slave'} at 127.0.0.1:54330
I20240501 07:59:30.789729 9619 tablet_replica.cc:330] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723: stopping tablet replica
I20240501 07:59:30.808576 9619 raft_consensus.cc:2229] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:30.808769 9619 raft_consensus.cc:2258] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:30.809679 9619 ts_tablet_manager.cc:1875] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:30.827384 9619 ts_tablet_manager.cc:1888] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.515
I20240501 07:59:30.827602 9619 log.cc:1196] T 114e9265cef4425799ad44e73e053b59 P f022e3632e9a428882a6cd16dc398723: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/114e9265cef4425799ad44e73e053b59
I20240501 07:59:30.828933 5573 catalog_manager.cc:4830] TS f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187): tablet 114e9265cef4425799ad44e73e053b59 (table rebalance_test_table_0 [id=3a5d5337264d45beb86597f2d8ffe305]) successfully deleted
I20240501 07:59:31.266505 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:31.273473 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:31.274904 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:31.275635 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:31.291966 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:31.303277 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:31.307968 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:31.699048 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:31.702955 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:31.708312 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:31.708515 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:31.733134 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:31.753630 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:31.761840 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:32.604287 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:32.627059 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:32.651463 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:32.660357 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:32.688535 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:32.738655 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:32.772470 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:33.308461 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:33.316545 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:33.325066 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:33.330459 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:33.349715 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:33.350015 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:33.359764 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:33.572546 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:33.572836 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:33.574961 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:33.575520 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:33.578639 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:33.591639 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:33.599974 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6ba367f2ecb14c299239fdf04770cb81 | 127.0.99.190:40255 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:39909 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
12f3f54871994efeb833126452c61da9 | 127.0.99.132:38041 | HEALTHY | <none> | 0 | 0
46c782f15fb643acb2ef72105f1fb04a | 127.0.99.129:33797 | HEALTHY | <none> | 2 | 0
72a601327d9d447aa7957cde6c36e000 | 127.0.99.133:34267 | HEALTHY | <none> | 1 | 0
a3af86b0e9a04ecbb15d305c48d7acb4 | 127.0.99.135:45001 | HEALTHY | <none> | 2 | 0
afc8c07c29354a04926027f57b8c6fab | 127.0.99.134:38017 | HEALTHY | <none> | 1 | 0
ec372f1e72b040e780e6bdd4d5409d33 | 127.0.99.131:38531 | HEALTHY | <none> | 5 | 0
f022e3632e9a428882a6cd16dc398723 | 127.0.99.130:42187 | HEALTHY | <none> | 5 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:33797
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:42187
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:38531
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:38041
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:34267
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:38017
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:45001
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:33797
server_dump_info_path | /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:42187
server_dump_info_path | /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:38531
server_dump_info_path | /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:38041
server_dump_info_path | /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:34267
server_dump_info_path | /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:38017
server_dump_info_path | /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:45001
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:39909 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 07:59:33.629446 398 log_verifier.cc:126] Checking tablet 0776eb886275492285da625d69e77e6b
I20240501 07:59:33.735643 398 log_verifier.cc:177] Verified matching terms for 599 ops in tablet 0776eb886275492285da625d69e77e6b
I20240501 07:59:33.735869 398 log_verifier.cc:126] Checking tablet 114e9265cef4425799ad44e73e053b59
I20240501 07:59:33.836627 398 log_verifier.cc:177] Verified matching terms for 600 ops in tablet 114e9265cef4425799ad44e73e053b59
I20240501 07:59:33.836858 398 log_verifier.cc:126] Checking tablet 1e15c1939984497fb1795ef7502545ca
I20240501 07:59:33.933914 398 log_verifier.cc:177] Verified matching terms for 573 ops in tablet 1e15c1939984497fb1795ef7502545ca
I20240501 07:59:33.934120 398 log_verifier.cc:126] Checking tablet 26ad4c1ac47544f586d3f46f5dc9c77e
I20240501 07:59:34.049634 398 log_verifier.cc:177] Verified matching terms for 700 ops in tablet 26ad4c1ac47544f586d3f46f5dc9c77e
I20240501 07:59:34.049865 398 log_verifier.cc:126] Checking tablet 2a577537e9294fd5bd8507232d1d1a76
I20240501 07:59:34.157418 398 log_verifier.cc:177] Verified matching terms for 663 ops in tablet 2a577537e9294fd5bd8507232d1d1a76
I20240501 07:59:34.157649 398 log_verifier.cc:126] Checking tablet 3f187bcb4c494805a8d6f366732b695a
I20240501 07:59:34.261081 398 log_verifier.cc:177] Verified matching terms for 640 ops in tablet 3f187bcb4c494805a8d6f366732b695a
I20240501 07:59:34.261299 398 log_verifier.cc:126] Checking tablet 65e9dbbdb1fa459fabc258f9e6878f00
I20240501 07:59:34.363818 398 log_verifier.cc:177] Verified matching terms for 619 ops in tablet 65e9dbbdb1fa459fabc258f9e6878f00
I20240501 07:59:34.364028 398 log_verifier.cc:126] Checking tablet 662d37043f054aa1a562cde8d6c155a9
I20240501 07:59:34.470515 398 log_verifier.cc:177] Verified matching terms for 636 ops in tablet 662d37043f054aa1a562cde8d6c155a9
I20240501 07:59:34.470739 398 log_verifier.cc:126] Checking tablet 7f4b2166aae54490a36ced8d37caf7b1
I20240501 07:59:34.572067 398 log_verifier.cc:177] Verified matching terms for 607 ops in tablet 7f4b2166aae54490a36ced8d37caf7b1
I20240501 07:59:34.572273 398 log_verifier.cc:126] Checking tablet 877dfd1e30ee4e12b16674d96d57c48e
I20240501 07:59:34.683230 398 log_verifier.cc:177] Verified matching terms for 661 ops in tablet 877dfd1e30ee4e12b16674d96d57c48e
I20240501 07:59:34.683452 398 log_verifier.cc:126] Checking tablet a07321bfc4c1445a906ec7c1261693e9
I20240501 07:59:34.778704 398 log_verifier.cc:177] Verified matching terms for 576 ops in tablet a07321bfc4c1445a906ec7c1261693e9
I20240501 07:59:34.778937 398 log_verifier.cc:126] Checking tablet a6cb1009e4fb4fb0a8e7d55c2c24ff65
I20240501 07:59:34.886632 398 log_verifier.cc:177] Verified matching terms for 649 ops in tablet a6cb1009e4fb4fb0a8e7d55c2c24ff65
I20240501 07:59:34.886879 398 log_verifier.cc:126] Checking tablet ba935cdc94cf4eeab62cfee9e789e2eb
I20240501 07:59:34.908321 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet ba935cdc94cf4eeab62cfee9e789e2eb
I20240501 07:59:34.908411 398 log_verifier.cc:126] Checking tablet bd95a7eff75845c0b49790c9657ef1e0
I20240501 07:59:35.033689 398 log_verifier.cc:177] Verified matching terms for 669 ops in tablet bd95a7eff75845c0b49790c9657ef1e0
I20240501 07:59:35.033898 398 log_verifier.cc:126] Checking tablet d26c0973a2dc4c639bc3a616e28fbfef
I20240501 07:59:35.137691 398 log_verifier.cc:177] Verified matching terms for 632 ops in tablet d26c0973a2dc4c639bc3a616e28fbfef
I20240501 07:59:35.137894 398 log_verifier.cc:126] Checking tablet dad26af7f84e42208fd4461656117035
I20240501 07:59:35.245793 398 log_verifier.cc:177] Verified matching terms for 625 ops in tablet dad26af7f84e42208fd4461656117035
I20240501 07:59:35.248471 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40255
--builtin_ntp_servers=127.0.99.148:39909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 07:59:35.371013 9853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:35.371263 9853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:35.371308 9853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:35.374706 9853 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:35.374791 9853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:35.374884 9853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240501 07:59:35.378831 9853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40255
--never_fsync=true
--heap_profile_path=/tmp/kudu.9853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:35.379812 9853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:35.380961 9853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:35.389938 9858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:35.390396 9861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:35.391281 9853 server_base.cc:1034] running on GCE node
W20240501 07:59:35.391664 9859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:35.392192 9853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:35.392900 9853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:35.394057 9853 hybrid_clock.cc:648] HybridClock initialized: now 1714550375394025 us; error 51 us; skew 500 ppm
I20240501 07:59:35.396302 9853 webserver.cc:460] Webserver started at http://127.0.99.136:33793/ using document root <none> and password file <none>
I20240501 07:59:35.396899 9853 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:35.396957 9853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:35.397171 9853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:35.399070 9853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a"
format_stamp: "Formatted at 2024-05-01 07:59:35 on dist-test-slave-2srn"
I20240501 07:59:35.399574 9853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a"
format_stamp: "Formatted at 2024-05-01 07:59:35 on dist-test-slave-2srn"
I20240501 07:59:35.403257 9853 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:59:35.405758 9867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:35.407078 9853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:35.407192 9853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a"
format_stamp: "Formatted at 2024-05-01 07:59:35 on dist-test-slave-2srn"
I20240501 07:59:35.407299 9853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:35.430951 9853 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:35.431685 9853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:35.431859 9853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:35.432474 9853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:35.433458 9853 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:35.433509 9853 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:35.433557 9853 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:35.433588 9853 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:35.444814 9853 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:43495
I20240501 07:59:35.444842 9980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:43495 every 8 connection(s)
I20240501 07:59:35.445847 9853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 07:59:35.446580 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 9853
I20240501 07:59:35.446705 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 07:59:35.456022 9981 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40255
I20240501 07:59:35.456288 9981 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:35.456830 9981 heartbeater.cc:511] Master 127.0.99.190:40255 requested a full tablet report, sending...
I20240501 07:59:35.457904 5580 ts_manager.cc:194] Registered new tserver with Master: abddb8eea0a34d4b9069ab7b89ff1e1a (127.0.99.136:43495)
I20240501 07:59:35.458576 5580 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:36445
I20240501 07:59:35.667730 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:35.668054 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:35.668597 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:35.671648 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:35.674602 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:35.683573 9915 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:35.692199 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:35.693079 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:35.799614 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:35.800477 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:35.803280 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:35.804196 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:35.805205 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:35.807035 9915 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:35.817950 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:35.822645 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:35.894311 5862 consensus_queue.cc:237] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 576, Committed index: 576, Last appended: 1.576, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 577 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:35.896255 7335 raft_consensus.cc:1261] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Refusing update from remote peer f022e3632e9a428882a6cd16dc398723: Log matching property violated. Preceding OpId in replica: term: 1 index: 576. Preceding OpId from leader: term: 1 index: 577. (index mismatch)
I20240501 07:59:35.896667 10086 consensus_queue.cc:1035] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 577, Last known committed idx: 576, Time since last communication: 0.000s
I20240501 07:59:35.898659 10086 raft_consensus.cc:2932] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Committing config change with OpId 1.577: config changed from index 228 to 577, VOTER abddb8eea0a34d4b9069ab7b89ff1e1a (127.0.99.136) added. New config: { opid_index: 577 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:35.899034 7335 raft_consensus.cc:2932] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Committing config change with OpId 1.577: config changed from index 228 to 577, VOTER abddb8eea0a34d4b9069ab7b89ff1e1a (127.0.99.136) added. New config: { opid_index: 577 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
W20240501 07:59:35.900563 5795 consensus_peers.cc:485] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 -> Peer abddb8eea0a34d4b9069ab7b89ff1e1a (127.0.99.136:43495): Couldn't send request to peer abddb8eea0a34d4b9069ab7b89ff1e1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a07321bfc4c1445a906ec7c1261693e9. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:35.901651 5588 catalog_manager.cc:5484] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 reported cstate change: config changed from index 228 to 577, VOTER abddb8eea0a34d4b9069ab7b89ff1e1a (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "f022e3632e9a428882a6cd16dc398723" committed_config { opid_index: 577 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:35.924501 7335 consensus_queue.cc:237] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 661, Committed index: 661, Last appended: 2.661, Last appended by leader: 175, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 662 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:35.926245 5730 raft_consensus.cc:1261] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a [term 2 FOLLOWER]: Refusing update from remote peer a3af86b0e9a04ecbb15d305c48d7acb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 661. Preceding OpId from leader: term: 2 index: 662. (index mismatch)
I20240501 07:59:35.926820 10109 consensus_queue.cc:1035] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 662, Last known committed idx: 661, Time since last communication: 0.000s
I20240501 07:59:35.928906 9852 raft_consensus.cc:2932] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [term 2 LEADER]: Committing config change with OpId 2.662: config changed from index 187 to 662, VOTER abddb8eea0a34d4b9069ab7b89ff1e1a (127.0.99.136) added. New config: { opid_index: 662 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
W20240501 07:59:35.930907 7269 consensus_peers.cc:485] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 -> Peer abddb8eea0a34d4b9069ab7b89ff1e1a (127.0.99.136:43495): Couldn't send request to peer abddb8eea0a34d4b9069ab7b89ff1e1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 877dfd1e30ee4e12b16674d96d57c48e. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:35.930868 5730 raft_consensus.cc:2932] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a [term 2 FOLLOWER]: Committing config change with OpId 2.662: config changed from index 187 to 662, VOTER abddb8eea0a34d4b9069ab7b89ff1e1a (127.0.99.136) added. New config: { opid_index: 662 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:35.933429 5583 catalog_manager.cc:5484] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a reported cstate change: config changed from index 187 to 662, VOTER abddb8eea0a34d4b9069ab7b89ff1e1a (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" committed_config { opid_index: 662 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:35.954890 5993 consensus_queue.cc:237] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 599, Committed index: 599, Last appended: 1.599, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 600 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:35.956830 7201 raft_consensus.cc:1261] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 1 index: 599. Preceding OpId from leader: term: 1 index: 600. (index mismatch)
I20240501 07:59:35.957371 10128 consensus_queue.cc:1035] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 600, Last known committed idx: 599, Time since last communication: 0.000s
W20240501 07:59:35.959815 5926 consensus_peers.cc:485] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 -> Peer abddb8eea0a34d4b9069ab7b89ff1e1a (127.0.99.136:43495): Couldn't send request to peer abddb8eea0a34d4b9069ab7b89ff1e1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0776eb886275492285da625d69e77e6b. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:35.959714 9711 raft_consensus.cc:2932] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Committing config change with OpId 1.600: config changed from index 298 to 600, VOTER abddb8eea0a34d4b9069ab7b89ff1e1a (127.0.99.136) added. New config: { opid_index: 600 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:35.961542 7201 raft_consensus.cc:2932] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Committing config change with OpId 1.600: config changed from index 298 to 600, VOTER abddb8eea0a34d4b9069ab7b89ff1e1a (127.0.99.136) added. New config: { opid_index: 600 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:35.962297 5588 catalog_manager.cc:5484] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 reported cstate change: config changed from index 298 to 600, VOTER abddb8eea0a34d4b9069ab7b89ff1e1a (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: 600 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:35.984148 5993 consensus_queue.cc:237] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 607, Committed index: 607, Last appended: 1.607, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 608 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:35.985597 6937 raft_consensus.cc:1261] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 1 index: 607. Preceding OpId from leader: term: 1 index: 608. (index mismatch)
W20240501 07:59:35.985766 5926 consensus_peers.cc:485] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 -> Peer abddb8eea0a34d4b9069ab7b89ff1e1a (127.0.99.136:43495): Couldn't send request to peer abddb8eea0a34d4b9069ab7b89ff1e1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f4b2166aae54490a36ced8d37caf7b1. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:35.985975 10128 consensus_queue.cc:1035] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 608, Last known committed idx: 607, Time since last communication: 0.000s
I20240501 07:59:35.987810 9711 raft_consensus.cc:2932] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Committing config change with OpId 1.608: config changed from index 238 to 608, VOTER abddb8eea0a34d4b9069ab7b89ff1e1a (127.0.99.136) added. New config: { opid_index: 608 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:35.988324 6937 raft_consensus.cc:2932] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Committing config change with OpId 1.608: config changed from index 238 to 608, VOTER abddb8eea0a34d4b9069ab7b89ff1e1a (127.0.99.136) added. New config: { opid_index: 608 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:35.990633 5583 catalog_manager.cc:5484] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 reported cstate change: config changed from index 238 to 608, VOTER abddb8eea0a34d4b9069ab7b89ff1e1a (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: 608 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:36.058166 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:36.059345 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:36.062963 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:36.067939 9915 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:36.074443 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:36.081965 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:36.082060 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:36.083324 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:36.172267 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:36.176666 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:36.182663 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:36.186522 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:36.186573 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:36.194777 9915 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:36.212802 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:36.212965 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:36.307731 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:36.309342 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:36.312472 9915 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:36.316167 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:36.317813 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:36.319100 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:36.319593 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:36.330610 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:36.373229 10263 ts_tablet_manager.cc:902] T 877dfd1e30ee4e12b16674d96d57c48e P abddb8eea0a34d4b9069ab7b89ff1e1a: Initiating tablet copy from peer a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135:45001)
I20240501 07:59:36.374446 10263 tablet_copy_client.cc:308] T 877dfd1e30ee4e12b16674d96d57c48e P abddb8eea0a34d4b9069ab7b89ff1e1a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:45001
I20240501 07:59:36.379082 7357 tablet_copy_service.cc:140] P a3af86b0e9a04ecbb15d305c48d7acb4: Received BeginTabletCopySession request for tablet 877dfd1e30ee4e12b16674d96d57c48e from peer abddb8eea0a34d4b9069ab7b89ff1e1a ({username='slave'} at 127.0.99.136:43257)
I20240501 07:59:36.379253 7357 tablet_copy_service.cc:161] P a3af86b0e9a04ecbb15d305c48d7acb4: Beginning new tablet copy session on tablet 877dfd1e30ee4e12b16674d96d57c48e from peer abddb8eea0a34d4b9069ab7b89ff1e1a at {username='slave'} at 127.0.99.136:43257: session id = abddb8eea0a34d4b9069ab7b89ff1e1a-877dfd1e30ee4e12b16674d96d57c48e
I20240501 07:59:36.380849 7357 tablet_copy_source_session.cc:205] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:36.384452 10263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 877dfd1e30ee4e12b16674d96d57c48e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:36.390151 10263 tablet_copy_client.cc:783] T 877dfd1e30ee4e12b16674d96d57c48e P abddb8eea0a34d4b9069ab7b89ff1e1a: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:36.390594 10263 tablet_copy_client.cc:647] T 877dfd1e30ee4e12b16674d96d57c48e P abddb8eea0a34d4b9069ab7b89ff1e1a: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:36.394865 10263 tablet_copy_client.cc:523] T 877dfd1e30ee4e12b16674d96d57c48e P abddb8eea0a34d4b9069ab7b89ff1e1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:36.397269 10263 tablet_bootstrap.cc:492] T 877dfd1e30ee4e12b16674d96d57c48e P abddb8eea0a34d4b9069ab7b89ff1e1a: Bootstrap starting.
I20240501 07:59:36.415601 10286 ts_tablet_manager.cc:902] T 7f4b2166aae54490a36ced8d37caf7b1 P abddb8eea0a34d4b9069ab7b89ff1e1a: Initiating tablet copy from peer ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531)
I20240501 07:59:36.416448 10286 tablet_copy_client.cc:308] T 7f4b2166aae54490a36ced8d37caf7b1 P abddb8eea0a34d4b9069ab7b89ff1e1a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38531
I20240501 07:59:36.423070 6013 tablet_copy_service.cc:140] P ec372f1e72b040e780e6bdd4d5409d33: Received BeginTabletCopySession request for tablet 7f4b2166aae54490a36ced8d37caf7b1 from peer abddb8eea0a34d4b9069ab7b89ff1e1a ({username='slave'} at 127.0.99.136:39895)
I20240501 07:59:36.423241 6013 tablet_copy_service.cc:161] P ec372f1e72b040e780e6bdd4d5409d33: Beginning new tablet copy session on tablet 7f4b2166aae54490a36ced8d37caf7b1 from peer abddb8eea0a34d4b9069ab7b89ff1e1a at {username='slave'} at 127.0.99.136:39895: session id = abddb8eea0a34d4b9069ab7b89ff1e1a-7f4b2166aae54490a36ced8d37caf7b1
I20240501 07:59:36.424208 6013 tablet_copy_source_session.cc:205] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:36.425359 10286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f4b2166aae54490a36ced8d37caf7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:36.427937 10286 tablet_copy_client.cc:783] T 7f4b2166aae54490a36ced8d37caf7b1 P abddb8eea0a34d4b9069ab7b89ff1e1a: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:36.428138 10286 tablet_copy_client.cc:647] T 7f4b2166aae54490a36ced8d37caf7b1 P abddb8eea0a34d4b9069ab7b89ff1e1a: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:36.431874 10286 tablet_copy_client.cc:523] T 7f4b2166aae54490a36ced8d37caf7b1 P abddb8eea0a34d4b9069ab7b89ff1e1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:36.433404 10286 tablet_bootstrap.cc:492] T 7f4b2166aae54490a36ced8d37caf7b1 P abddb8eea0a34d4b9069ab7b89ff1e1a: Bootstrap starting.
I20240501 07:59:36.437199 10296 ts_tablet_manager.cc:902] T 0776eb886275492285da625d69e77e6b P abddb8eea0a34d4b9069ab7b89ff1e1a: Initiating tablet copy from peer ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531)
I20240501 07:59:36.437919 10296 tablet_copy_client.cc:308] T 0776eb886275492285da625d69e77e6b P abddb8eea0a34d4b9069ab7b89ff1e1a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38531
I20240501 07:59:36.438670 6013 tablet_copy_service.cc:140] P ec372f1e72b040e780e6bdd4d5409d33: Received BeginTabletCopySession request for tablet 0776eb886275492285da625d69e77e6b from peer abddb8eea0a34d4b9069ab7b89ff1e1a ({username='slave'} at 127.0.99.136:39895)
I20240501 07:59:36.438795 6013 tablet_copy_service.cc:161] P ec372f1e72b040e780e6bdd4d5409d33: Beginning new tablet copy session on tablet 0776eb886275492285da625d69e77e6b from peer abddb8eea0a34d4b9069ab7b89ff1e1a at {username='slave'} at 127.0.99.136:39895: session id = abddb8eea0a34d4b9069ab7b89ff1e1a-0776eb886275492285da625d69e77e6b
I20240501 07:59:36.439808 6013 tablet_copy_source_session.cc:205] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:36.440935 10296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0776eb886275492285da625d69e77e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:36.444242 10296 tablet_copy_client.cc:783] T 0776eb886275492285da625d69e77e6b P abddb8eea0a34d4b9069ab7b89ff1e1a: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:36.444538 10296 tablet_copy_client.cc:647] T 0776eb886275492285da625d69e77e6b P abddb8eea0a34d4b9069ab7b89ff1e1a: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:36.458137 10263 log.cc:824] T 877dfd1e30ee4e12b16674d96d57c48e P abddb8eea0a34d4b9069ab7b89ff1e1a: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:36.460704 9981 heartbeater.cc:503] Master 127.0.99.190:40255 was elected leader, sending a full tablet report...
I20240501 07:59:36.462728 10309 ts_tablet_manager.cc:902] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a: Initiating tablet copy from peer f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187)
I20240501 07:59:36.463325 10309 tablet_copy_client.cc:308] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42187
I20240501 07:59:36.463991 10296 tablet_copy_client.cc:523] T 0776eb886275492285da625d69e77e6b P abddb8eea0a34d4b9069ab7b89ff1e1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:36.465579 10296 tablet_bootstrap.cc:492] T 0776eb886275492285da625d69e77e6b P abddb8eea0a34d4b9069ab7b89ff1e1a: Bootstrap starting.
I20240501 07:59:36.471942 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:36.475518 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:36.478885 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:36.479171 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:36.489596 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:36.504027 5882 tablet_copy_service.cc:140] P f022e3632e9a428882a6cd16dc398723: Received BeginTabletCopySession request for tablet a07321bfc4c1445a906ec7c1261693e9 from peer abddb8eea0a34d4b9069ab7b89ff1e1a ({username='slave'} at 127.0.99.136:34705)
I20240501 07:59:36.504184 5882 tablet_copy_service.cc:161] P f022e3632e9a428882a6cd16dc398723: Beginning new tablet copy session on tablet a07321bfc4c1445a906ec7c1261693e9 from peer abddb8eea0a34d4b9069ab7b89ff1e1a at {username='slave'} at 127.0.99.136:34705: session id = abddb8eea0a34d4b9069ab7b89ff1e1a-a07321bfc4c1445a906ec7c1261693e9
I20240501 07:59:36.505044 5882 tablet_copy_source_session.cc:205] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:36.506073 10309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a07321bfc4c1445a906ec7c1261693e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:36.509692 10309 tablet_copy_client.cc:783] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:36.509881 10309 tablet_copy_client.cc:647] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:36.515022 9915 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:36.516199 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:36.526762 10309 tablet_copy_client.cc:523] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:36.528254 10309 tablet_bootstrap.cc:492] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a: Bootstrap starting.
I20240501 07:59:36.538229 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:36.709877 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:36.710187 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:36.720235 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:36.722538 9915 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:36.732985 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:36.742722 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:36.780620 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:36.795542 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:37.027513 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:37.032444 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:37.037425 9915 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:37.055827 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:37.081915 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:37.087184 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:37.098047 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:37.115408 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:37.140856 10286 tablet_bootstrap.cc:492] T 7f4b2166aae54490a36ced8d37caf7b1 P abddb8eea0a34d4b9069ab7b89ff1e1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=608 overwritten=0 applied=608 ignored=0} inserts{seen=1237 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:37.141474 10286 tablet_bootstrap.cc:492] T 7f4b2166aae54490a36ced8d37caf7b1 P abddb8eea0a34d4b9069ab7b89ff1e1a: Bootstrap complete.
I20240501 07:59:37.141996 10286 ts_tablet_manager.cc:1367] T 7f4b2166aae54490a36ced8d37caf7b1 P abddb8eea0a34d4b9069ab7b89ff1e1a: Time spent bootstrapping tablet: real 0.709s user 0.425s sys 0.024s
I20240501 07:59:37.145006 10286 raft_consensus.cc:348] T 7f4b2166aae54490a36ced8d37caf7b1 P abddb8eea0a34d4b9069ab7b89ff1e1a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 608 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:37.145486 10286 raft_consensus.cc:729] T 7f4b2166aae54490a36ced8d37caf7b1 P abddb8eea0a34d4b9069ab7b89ff1e1a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: abddb8eea0a34d4b9069ab7b89ff1e1a, State: Initialized, Role: FOLLOWER
I20240501 07:59:37.146090 10286 consensus_queue.cc:260] T 7f4b2166aae54490a36ced8d37caf7b1 P abddb8eea0a34d4b9069ab7b89ff1e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 608, Last appended: 1.608, Last appended by leader: 608, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 608 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:37.147382 10286 ts_tablet_manager.cc:1398] T 7f4b2166aae54490a36ced8d37caf7b1 P abddb8eea0a34d4b9069ab7b89ff1e1a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 07:59:37.150462 6013 tablet_copy_service.cc:342] P ec372f1e72b040e780e6bdd4d5409d33: Request end of tablet copy session abddb8eea0a34d4b9069ab7b89ff1e1a-7f4b2166aae54490a36ced8d37caf7b1 received from {username='slave'} at 127.0.99.136:39895
I20240501 07:59:37.150575 6013 tablet_copy_service.cc:433] P ec372f1e72b040e780e6bdd4d5409d33: ending tablet copy session abddb8eea0a34d4b9069ab7b89ff1e1a-7f4b2166aae54490a36ced8d37caf7b1 on tablet 7f4b2166aae54490a36ced8d37caf7b1 with peer abddb8eea0a34d4b9069ab7b89ff1e1a
I20240501 07:59:37.162240 10263 tablet_bootstrap.cc:492] T 877dfd1e30ee4e12b16674d96d57c48e P abddb8eea0a34d4b9069ab7b89ff1e1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=662 overwritten=0 applied=662 ignored=0} inserts{seen=1344 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:37.162642 10263 tablet_bootstrap.cc:492] T 877dfd1e30ee4e12b16674d96d57c48e P abddb8eea0a34d4b9069ab7b89ff1e1a: Bootstrap complete.
I20240501 07:59:37.162925 10263 ts_tablet_manager.cc:1367] T 877dfd1e30ee4e12b16674d96d57c48e P abddb8eea0a34d4b9069ab7b89ff1e1a: Time spent bootstrapping tablet: real 0.766s user 0.445s sys 0.048s
I20240501 07:59:37.163632 10263 raft_consensus.cc:348] T 877dfd1e30ee4e12b16674d96d57c48e P abddb8eea0a34d4b9069ab7b89ff1e1a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 662 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:37.163765 10263 raft_consensus.cc:729] T 877dfd1e30ee4e12b16674d96d57c48e P abddb8eea0a34d4b9069ab7b89ff1e1a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: abddb8eea0a34d4b9069ab7b89ff1e1a, State: Initialized, Role: FOLLOWER
I20240501 07:59:37.163966 10263 consensus_queue.cc:260] T 877dfd1e30ee4e12b16674d96d57c48e P abddb8eea0a34d4b9069ab7b89ff1e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 662, Last appended: 2.662, Last appended by leader: 662, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 662 OBSOLETE_local: false peers { permanent_uuid: "46c782f15fb643acb2ef72105f1fb04a" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33797 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:37.164394 10263 ts_tablet_manager.cc:1398] T 877dfd1e30ee4e12b16674d96d57c48e P abddb8eea0a34d4b9069ab7b89ff1e1a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:37.165196 7357 tablet_copy_service.cc:342] P a3af86b0e9a04ecbb15d305c48d7acb4: Request end of tablet copy session abddb8eea0a34d4b9069ab7b89ff1e1a-877dfd1e30ee4e12b16674d96d57c48e received from {username='slave'} at 127.0.99.136:43257
I20240501 07:59:37.165308 7357 tablet_copy_service.cc:433] P a3af86b0e9a04ecbb15d305c48d7acb4: ending tablet copy session abddb8eea0a34d4b9069ab7b89ff1e1a-877dfd1e30ee4e12b16674d96d57c48e on tablet 877dfd1e30ee4e12b16674d96d57c48e with peer abddb8eea0a34d4b9069ab7b89ff1e1a
I20240501 07:59:37.188218 10309 tablet_bootstrap.cc:492] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=577 overwritten=0 applied=577 ignored=0} inserts{seen=1247 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:37.188652 10309 tablet_bootstrap.cc:492] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a: Bootstrap complete.
I20240501 07:59:37.188959 10309 ts_tablet_manager.cc:1367] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a: Time spent bootstrapping tablet: real 0.661s user 0.406s sys 0.020s
I20240501 07:59:37.189558 10309 raft_consensus.cc:348] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 577 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:37.189772 10309 raft_consensus.cc:729] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: abddb8eea0a34d4b9069ab7b89ff1e1a, State: Initialized, Role: FOLLOWER
I20240501 07:59:37.190007 10309 consensus_queue.cc:260] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 577, Last appended: 1.577, Last appended by leader: 577, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 577 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:37.190608 10309 ts_tablet_manager.cc:1398] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:37.191161 5882 tablet_copy_service.cc:342] P f022e3632e9a428882a6cd16dc398723: Request end of tablet copy session abddb8eea0a34d4b9069ab7b89ff1e1a-a07321bfc4c1445a906ec7c1261693e9 received from {username='slave'} at 127.0.99.136:34705
I20240501 07:59:37.191243 5882 tablet_copy_service.cc:433] P f022e3632e9a428882a6cd16dc398723: ending tablet copy session abddb8eea0a34d4b9069ab7b89ff1e1a-a07321bfc4c1445a906ec7c1261693e9 on tablet a07321bfc4c1445a906ec7c1261693e9 with peer abddb8eea0a34d4b9069ab7b89ff1e1a
I20240501 07:59:37.201179 10296 tablet_bootstrap.cc:492] T 0776eb886275492285da625d69e77e6b P abddb8eea0a34d4b9069ab7b89ff1e1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=600 overwritten=0 applied=600 ignored=0} inserts{seen=1229 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:37.201462 10296 tablet_bootstrap.cc:492] T 0776eb886275492285da625d69e77e6b P abddb8eea0a34d4b9069ab7b89ff1e1a: Bootstrap complete.
I20240501 07:59:37.201670 10296 ts_tablet_manager.cc:1367] T 0776eb886275492285da625d69e77e6b P abddb8eea0a34d4b9069ab7b89ff1e1a: Time spent bootstrapping tablet: real 0.736s user 0.398s sys 0.035s
I20240501 07:59:37.202098 10296 raft_consensus.cc:348] T 0776eb886275492285da625d69e77e6b P abddb8eea0a34d4b9069ab7b89ff1e1a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 600 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:37.202199 10296 raft_consensus.cc:729] T 0776eb886275492285da625d69e77e6b P abddb8eea0a34d4b9069ab7b89ff1e1a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: abddb8eea0a34d4b9069ab7b89ff1e1a, State: Initialized, Role: FOLLOWER
I20240501 07:59:37.202320 10296 consensus_queue.cc:260] T 0776eb886275492285da625d69e77e6b P abddb8eea0a34d4b9069ab7b89ff1e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 600, Last appended: 1.600, Last appended by leader: 600, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 600 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "afc8c07c29354a04926027f57b8c6fab" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38017 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:37.202569 10296 ts_tablet_manager.cc:1398] T 0776eb886275492285da625d69e77e6b P abddb8eea0a34d4b9069ab7b89ff1e1a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 07:59:37.203135 6013 tablet_copy_service.cc:342] P ec372f1e72b040e780e6bdd4d5409d33: Request end of tablet copy session abddb8eea0a34d4b9069ab7b89ff1e1a-0776eb886275492285da625d69e77e6b received from {username='slave'} at 127.0.99.136:39895
I20240501 07:59:37.203215 6013 tablet_copy_service.cc:433] P ec372f1e72b040e780e6bdd4d5409d33: ending tablet copy session abddb8eea0a34d4b9069ab7b89ff1e1a-0776eb886275492285da625d69e77e6b on tablet 0776eb886275492285da625d69e77e6b with peer abddb8eea0a34d4b9069ab7b89ff1e1a
I20240501 07:59:37.267597 9935 raft_consensus.cc:1203] T 877dfd1e30ee4e12b16674d96d57c48e P abddb8eea0a34d4b9069ab7b89ff1e1a [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.661->[2.662-2.662] Dedup: 2.662->[]
I20240501 07:59:37.331338 9935 raft_consensus.cc:1203] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.576->[1.577-1.577] Dedup: 1.577->[]
I20240501 07:59:37.424829 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:37.425776 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:37.427253 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:37.427349 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:37.430295 9915 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:37.435839 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:37.448264 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:37.459408 9935 raft_consensus.cc:1203] T 7f4b2166aae54490a36ced8d37caf7b1 P abddb8eea0a34d4b9069ab7b89ff1e1a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.607->[1.608-1.608] Dedup: 1.608->[]
I20240501 07:59:37.464409 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:37.548727 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:37.553210 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:37.553475 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:37.554867 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:37.559427 9915 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:37.559649 9935 raft_consensus.cc:1203] T 0776eb886275492285da625d69e77e6b P abddb8eea0a34d4b9069ab7b89ff1e1a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.599->[1.600-1.600] Dedup: 1.600->[]
I20240501 07:59:37.566121 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:37.571028 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:37.592720 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:37.680248 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:37.680521 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:37.682852 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:37.684198 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:37.691648 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:37.698369 9915 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:37.709283 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:37.727347 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:37.763862 5993 consensus_queue.cc:237] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 600, Committed index: 600, Last appended: 1.600, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 601 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:37.765125 9935 raft_consensus.cc:1261] T 0776eb886275492285da625d69e77e6b P abddb8eea0a34d4b9069ab7b89ff1e1a [term 1 FOLLOWER]: Refusing update from remote peer ec372f1e72b040e780e6bdd4d5409d33: Log matching property violated. Preceding OpId in replica: term: 1 index: 600. Preceding OpId from leader: term: 1 index: 601. (index mismatch)
I20240501 07:59:37.765532 10128 consensus_queue.cc:1035] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 601, Last known committed idx: 600, Time since last communication: 0.000s
I20240501 07:59:37.768139 10528 raft_consensus.cc:2932] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Committing config change with OpId 1.601: config changed from index 600 to 601, VOTER afc8c07c29354a04926027f57b8c6fab (127.0.99.134) evicted. New config: { opid_index: 601 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:37.768682 9935 raft_consensus.cc:2932] T 0776eb886275492285da625d69e77e6b P abddb8eea0a34d4b9069ab7b89ff1e1a [term 1 FOLLOWER]: Committing config change with OpId 1.601: config changed from index 600 to 601, VOTER afc8c07c29354a04926027f57b8c6fab (127.0.99.134) evicted. New config: { opid_index: 601 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:37.771106 5588 catalog_manager.cc:5484] T 0776eb886275492285da625d69e77e6b P ec372f1e72b040e780e6bdd4d5409d33 reported cstate change: config changed from index 600 to 601, VOTER afc8c07c29354a04926027f57b8c6fab (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "ec372f1e72b040e780e6bdd4d5409d33" committed_config { opid_index: 601 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:37.777942 7170 tablet_service.cc:1514] Processing DeleteTablet for tablet 0776eb886275492285da625d69e77e6b with delete_type TABLET_DATA_TOMBSTONED (TS afc8c07c29354a04926027f57b8c6fab not found in new config with opid_index 601) from {username='slave'} at 127.0.0.1:40376
I20240501 07:59:37.778503 10539 tablet_replica.cc:330] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab: stopping tablet replica
I20240501 07:59:37.778800 10539 raft_consensus.cc:2229] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:37.779045 10539 raft_consensus.cc:2258] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:37.780611 10539 ts_tablet_manager.cc:1875] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:37.783069 10539 ts_tablet_manager.cc:1888] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.600
I20240501 07:59:37.783231 10539 log.cc:1196] T 0776eb886275492285da625d69e77e6b P afc8c07c29354a04926027f57b8c6fab: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/0776eb886275492285da625d69e77e6b
I20240501 07:59:37.783860 5573 catalog_manager.cc:4830] TS afc8c07c29354a04926027f57b8c6fab (127.0.99.134:38017): tablet 0776eb886275492285da625d69e77e6b (table rebalance_test_table_3 [id=62d90ece49d2427fbd192ff552b40579]) successfully deleted
I20240501 07:59:37.836681 9915 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:37.840516 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:37.843341 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:37.843346 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:37.851078 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:37.853459 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:37.872834 5841 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:37.874128 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:37.913268 5862 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a07321bfc4c1445a906ec7c1261693e9"
dest_uuid: "f022e3632e9a428882a6cd16dc398723"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58242
I20240501 07:59:37.913436 5862 raft_consensus.cc:595] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:37.954543 10400 raft_consensus.cc:979] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723: : Instructing follower abddb8eea0a34d4b9069ab7b89ff1e1a to start an election
I20240501 07:59:37.954668 10086 raft_consensus.cc:1067] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Signalling peer abddb8eea0a34d4b9069ab7b89ff1e1a to start an election
I20240501 07:59:37.955158 9935 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a07321bfc4c1445a906ec7c1261693e9"
dest_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a"
from {username='slave'} at 127.0.99.130:37953
I20240501 07:59:37.955374 9935 raft_consensus.cc:482] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:37.955490 9935 raft_consensus.cc:3037] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.956476 9935 raft_consensus.cc:504] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 577 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:37.957677 9935 leader_election.cc:288] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a [CANDIDATE]: Term 2 election: Requested vote from peers f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187), a3af86b0e9a04ecbb15d305c48d7acb4 (127.0.99.135:45001)
I20240501 07:59:37.958299 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07321bfc4c1445a906ec7c1261693e9" candidate_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 577 } } ignore_live_leader: true dest_uuid: "f022e3632e9a428882a6cd16dc398723"
I20240501 07:59:37.958463 5862 raft_consensus.cc:3032] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:37.958508 5862 raft_consensus.cc:729] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f022e3632e9a428882a6cd16dc398723, State: Running, Role: LEADER
I20240501 07:59:37.958498 7335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07321bfc4c1445a906ec7c1261693e9" candidate_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 577 } } ignore_live_leader: true dest_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4"
I20240501 07:59:37.958607 7335 raft_consensus.cc:3037] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.958664 5862 consensus_queue.cc:260] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 577, Committed index: 577, Last appended: 1.577, Last appended by leader: 577, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 577 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:37.958814 5862 raft_consensus.cc:3037] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:37.959697 5862 raft_consensus.cc:2454] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abddb8eea0a34d4b9069ab7b89ff1e1a in term 2.
I20240501 07:59:37.959697 7335 raft_consensus.cc:2454] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abddb8eea0a34d4b9069ab7b89ff1e1a in term 2.
I20240501 07:59:37.960294 9868 leader_election.cc:302] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abddb8eea0a34d4b9069ab7b89ff1e1a, f022e3632e9a428882a6cd16dc398723; no voters:
I20240501 07:59:37.960538 10398 raft_consensus.cc:2781] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:37.960760 10398 raft_consensus.cc:686] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a [term 2 LEADER]: Becoming Leader. State: Replica: abddb8eea0a34d4b9069ab7b89ff1e1a, State: Running, Role: LEADER
I20240501 07:59:37.961050 10398 consensus_queue.cc:237] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 577, Committed index: 577, Last appended: 1.577, Last appended by leader: 577, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 577 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:37.963341 5589 catalog_manager.cc:5484] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a reported cstate change: term changed from 1 to 2, leader changed from f022e3632e9a428882a6cd16dc398723 (127.0.99.130) to abddb8eea0a34d4b9069ab7b89ff1e1a (127.0.99.136). New cstate: current_term: 2 leader_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" committed_config { opid_index: 577 OBSOLETE_local: false peers { permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 } } peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:37.975071 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:37.981598 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:37.982718 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:37.983887 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:37.983940 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:37.989554 9915 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:38.001740 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:38.010692 5841 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:38.104949 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:38.106393 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:38.108055 9915 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:38.109256 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:38.119850 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:38.121825 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:38.125118 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:38.132460 5841 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:38.186864 7335 consensus_queue.cc:237] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 662, Committed index: 662, Last appended: 2.662, Last appended by leader: 175, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 663 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:38.188162 9935 raft_consensus.cc:1261] T 877dfd1e30ee4e12b16674d96d57c48e P abddb8eea0a34d4b9069ab7b89ff1e1a [term 2 FOLLOWER]: Refusing update from remote peer a3af86b0e9a04ecbb15d305c48d7acb4: Log matching property violated. Preceding OpId in replica: term: 2 index: 662. Preceding OpId from leader: term: 2 index: 663. (index mismatch)
I20240501 07:59:38.188587 10109 consensus_queue.cc:1035] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 663, Last known committed idx: 662, Time since last communication: 0.000s
I20240501 07:59:38.190450 10670 raft_consensus.cc:2932] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 [term 2 LEADER]: Committing config change with OpId 2.663: config changed from index 662 to 663, VOTER 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129) evicted. New config: { opid_index: 663 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:38.191068 9935 raft_consensus.cc:2932] T 877dfd1e30ee4e12b16674d96d57c48e P abddb8eea0a34d4b9069ab7b89ff1e1a [term 2 FOLLOWER]: Committing config change with OpId 2.663: config changed from index 662 to 663, VOTER 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129) evicted. New config: { opid_index: 663 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:38.193665 5585 catalog_manager.cc:5484] T 877dfd1e30ee4e12b16674d96d57c48e P a3af86b0e9a04ecbb15d305c48d7acb4 reported cstate change: config changed from index 662 to 663, VOTER 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" committed_config { opid_index: 663 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:38.196664 5710 tablet_service.cc:1514] Processing DeleteTablet for tablet 877dfd1e30ee4e12b16674d96d57c48e with delete_type TABLET_DATA_TOMBSTONED (TS 46c782f15fb643acb2ef72105f1fb04a not found in new config with opid_index 663) from {username='slave'} at 127.0.0.1:48480
I20240501 07:59:38.197052 10671 tablet_replica.cc:330] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a: stopping tablet replica
I20240501 07:59:38.197201 10671 raft_consensus.cc:2229] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:38.197338 10671 raft_consensus.cc:2258] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:38.198640 10671 ts_tablet_manager.cc:1875] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:38.200517 10671 ts_tablet_manager.cc:1888] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.662
I20240501 07:59:38.200592 10671 log.cc:1196] T 877dfd1e30ee4e12b16674d96d57c48e P 46c782f15fb643acb2ef72105f1fb04a: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/877dfd1e30ee4e12b16674d96d57c48e
I20240501 07:59:38.201159 5575 catalog_manager.cc:4830] TS 46c782f15fb643acb2ef72105f1fb04a (127.0.99.129:33797): tablet 877dfd1e30ee4e12b16674d96d57c48e (table rebalance_test_table_4 [id=2c9dba3a010e4d3fa551b89730285ff8]) successfully deleted
I20240501 07:59:38.462723 5862 raft_consensus.cc:1261] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Refusing update from remote peer abddb8eea0a34d4b9069ab7b89ff1e1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 577. Preceding OpId from leader: term: 2 index: 578. (index mismatch)
I20240501 07:59:38.463490 10398 consensus_queue.cc:1035] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f022e3632e9a428882a6cd16dc398723" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 578, Last known committed idx: 577, Time since last communication: 0.000s
I20240501 07:59:38.466971 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:38.468528 7335 raft_consensus.cc:1261] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 2 FOLLOWER]: Refusing update from remote peer abddb8eea0a34d4b9069ab7b89ff1e1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 577. Preceding OpId from leader: term: 2 index: 578. (index mismatch)
I20240501 07:59:38.468750 9915 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:38.469933 10398 consensus_queue.cc:1035] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 578, Last known committed idx: 577, Time since last communication: 0.000s
I20240501 07:59:38.472671 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:38.476521 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:38.485280 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:38.486451 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:38.504117 5841 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:38.504851 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:38.532518 5993 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7f4b2166aae54490a36ced8d37caf7b1"
dest_uuid: "ec372f1e72b040e780e6bdd4d5409d33"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53852
I20240501 07:59:38.532685 5993 raft_consensus.cc:595] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:38.587127 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:38.588342 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:38.594085 6920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:38.595194 5972 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:38.600608 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:38.603896 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:38.618723 9915 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:38.626654 10528 raft_consensus.cc:979] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33: : Instructing follower 12f3f54871994efeb833126452c61da9 to start an election
I20240501 07:59:38.626793 10528 raft_consensus.cc:1067] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Signalling peer 12f3f54871994efeb833126452c61da9 to start an election
I20240501 07:59:38.627290 6937 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7f4b2166aae54490a36ced8d37caf7b1"
dest_uuid: "12f3f54871994efeb833126452c61da9"
from {username='slave'} at 127.0.99.131:52469
I20240501 07:59:38.627516 6937 raft_consensus.cc:482] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:38.627574 6937 raft_consensus.cc:3037] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:38.628645 6937 raft_consensus.cc:504] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 608 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:38.629793 6937 leader_election.cc:288] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [CANDIDATE]: Term 2 election: Requested vote from peers ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531), abddb8eea0a34d4b9069ab7b89ff1e1a (127.0.99.136:43495)
I20240501 07:59:38.631983 5993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b2166aae54490a36ced8d37caf7b1" candidate_uuid: "12f3f54871994efeb833126452c61da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 608 } } ignore_live_leader: true dest_uuid: "ec372f1e72b040e780e6bdd4d5409d33"
I20240501 07:59:38.632133 5993 raft_consensus.cc:3032] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:38.632179 5993 raft_consensus.cc:729] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec372f1e72b040e780e6bdd4d5409d33, State: Running, Role: LEADER
I20240501 07:59:38.632329 5993 consensus_queue.cc:260] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 608, Committed index: 608, Last appended: 1.608, Last appended by leader: 608, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 608 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:38.632517 5993 raft_consensus.cc:3037] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:38.633628 5993 raft_consensus.cc:2454] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12f3f54871994efeb833126452c61da9 in term 2.
I20240501 07:59:38.634042 6875 leader_election.cc:302] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12f3f54871994efeb833126452c61da9, ec372f1e72b040e780e6bdd4d5409d33; no voters:
I20240501 07:59:38.638371 5841 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:38.640316 9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f4b2166aae54490a36ced8d37caf7b1" candidate_uuid: "12f3f54871994efeb833126452c61da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 608 } } ignore_live_leader: true dest_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a"
I20240501 07:59:38.640537 9935 raft_consensus.cc:3037] T 7f4b2166aae54490a36ced8d37caf7b1 P abddb8eea0a34d4b9069ab7b89ff1e1a [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:38.641608 9935 raft_consensus.cc:2454] T 7f4b2166aae54490a36ced8d37caf7b1 P abddb8eea0a34d4b9069ab7b89ff1e1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12f3f54871994efeb833126452c61da9 in term 2.
I20240501 07:59:38.650858 10756 raft_consensus.cc:2781] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:38.652159 10756 raft_consensus.cc:686] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [term 2 LEADER]: Becoming Leader. State: Replica: 12f3f54871994efeb833126452c61da9, State: Running, Role: LEADER
I20240501 07:59:38.652549 10756 consensus_queue.cc:237] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 608, Committed index: 608, Last appended: 1.608, Last appended by leader: 608, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 608 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:38.655167 5586 catalog_manager.cc:5484] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 reported cstate change: term changed from 1 to 2, leader changed from ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) to 12f3f54871994efeb833126452c61da9 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "12f3f54871994efeb833126452c61da9" committed_config { opid_index: 608 OBSOLETE_local: false peers { permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 } } peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:38.743448 6920 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:38.744503 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:38.745052 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:38.750181 5972 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:38.756099 9915 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:38.757521 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:38.759449 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:38.780750 5841 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:38.824234 9935 consensus_queue.cc:237] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 578, Committed index: 578, Last appended: 2.578, Last appended by leader: 577, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 579 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:38.825750 7335 raft_consensus.cc:1261] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 2 FOLLOWER]: Refusing update from remote peer abddb8eea0a34d4b9069ab7b89ff1e1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 578. Preceding OpId from leader: term: 2 index: 579. (index mismatch)
I20240501 07:59:38.826534 10711 consensus_queue.cc:1035] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 579, Last known committed idx: 578, Time since last communication: 0.000s
I20240501 07:59:38.828987 10398 raft_consensus.cc:2932] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a [term 2 LEADER]: Committing config change with OpId 2.579: config changed from index 577 to 579, VOTER f022e3632e9a428882a6cd16dc398723 (127.0.99.130) evicted. New config: { opid_index: 579 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:38.829614 7335 raft_consensus.cc:2932] T a07321bfc4c1445a906ec7c1261693e9 P a3af86b0e9a04ecbb15d305c48d7acb4 [term 2 FOLLOWER]: Committing config change with OpId 2.579: config changed from index 577 to 579, VOTER f022e3632e9a428882a6cd16dc398723 (127.0.99.130) evicted. New config: { opid_index: 579 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:38.832216 5585 catalog_manager.cc:5484] T a07321bfc4c1445a906ec7c1261693e9 P abddb8eea0a34d4b9069ab7b89ff1e1a reported cstate change: config changed from index 577 to 579, VOTER f022e3632e9a428882a6cd16dc398723 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" committed_config { opid_index: 579 OBSOLETE_local: false peers { permanent_uuid: "a3af86b0e9a04ecbb15d305c48d7acb4" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45001 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:38.835919 5841 tablet_service.cc:1514] Processing DeleteTablet for tablet a07321bfc4c1445a906ec7c1261693e9 with delete_type TABLET_DATA_TOMBSTONED (TS f022e3632e9a428882a6cd16dc398723 not found in new config with opid_index 579) from {username='slave'} at 127.0.0.1:54330
I20240501 07:59:38.836360 10834 tablet_replica.cc:330] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723: stopping tablet replica
I20240501 07:59:38.836499 10834 raft_consensus.cc:2229] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:38.836611 10834 raft_consensus.cc:2258] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:38.837896 10834 ts_tablet_manager.cc:1875] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:38.840677 10834 ts_tablet_manager.cc:1888] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.578
I20240501 07:59:38.840770 10834 log.cc:1196] T a07321bfc4c1445a906ec7c1261693e9 P f022e3632e9a428882a6cd16dc398723: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a07321bfc4c1445a906ec7c1261693e9
I20240501 07:59:38.841470 5573 catalog_manager.cc:4830] TS f022e3632e9a428882a6cd16dc398723 (127.0.99.130:42187): tablet a07321bfc4c1445a906ec7c1261693e9 (table rebalance_test_table_1 [id=b2c8da5392b24236b9cc0f5c750c0528]) successfully deleted
I20240501 07:59:38.905180 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:38.905481 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:38.910460 6920 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:38.913520 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:38.914090 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:38.920511 5841 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:38.928560 5972 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:38.934788 9915 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.029446 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:39.029886 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.036217 6920 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.037099 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:39.046530 9915 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.051216 5972 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:39.058913 5841 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:39.071225 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.122815 5993 raft_consensus.cc:1261] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 2 FOLLOWER]: Refusing update from remote peer 12f3f54871994efeb833126452c61da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 608. Preceding OpId from leader: term: 2 index: 609. (index mismatch)
I20240501 07:59:39.123420 10757 consensus_queue.cc:1035] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec372f1e72b040e780e6bdd4d5409d33" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 609, Last known committed idx: 608, Time since last communication: 0.000s
I20240501 07:59:39.126577 9935 raft_consensus.cc:1261] T 7f4b2166aae54490a36ced8d37caf7b1 P abddb8eea0a34d4b9069ab7b89ff1e1a [term 2 FOLLOWER]: Refusing update from remote peer 12f3f54871994efeb833126452c61da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 608. Preceding OpId from leader: term: 2 index: 609. (index mismatch)
I20240501 07:59:39.126981 10756 consensus_queue.cc:1035] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 609, Last known committed idx: 608, Time since last communication: 0.000s
I20240501 07:59:39.354343 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.360687 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:39.361737 6920 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.362138 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:39.369273 9915 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.369668 5841 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:39.380317 5972 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:39.382814 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.433744 6937 consensus_queue.cc:237] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 609, Committed index: 609, Last appended: 2.609, Last appended by leader: 608, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 610 OBSOLETE_local: false peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } }
I20240501 07:59:39.434988 9935 raft_consensus.cc:1261] T 7f4b2166aae54490a36ced8d37caf7b1 P abddb8eea0a34d4b9069ab7b89ff1e1a [term 2 FOLLOWER]: Refusing update from remote peer 12f3f54871994efeb833126452c61da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 609. Preceding OpId from leader: term: 2 index: 610. (index mismatch)
I20240501 07:59:39.435413 10756 consensus_queue.cc:1035] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 610, Last known committed idx: 609, Time since last communication: 0.000s
I20240501 07:59:39.436996 10757 raft_consensus.cc:2932] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 [term 2 LEADER]: Committing config change with OpId 2.610: config changed from index 608 to 610, VOTER ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) evicted. New config: { opid_index: 610 OBSOLETE_local: false peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:39.437460 9935 raft_consensus.cc:2932] T 7f4b2166aae54490a36ced8d37caf7b1 P abddb8eea0a34d4b9069ab7b89ff1e1a [term 2 FOLLOWER]: Committing config change with OpId 2.610: config changed from index 608 to 610, VOTER ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) evicted. New config: { opid_index: 610 OBSOLETE_local: false peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:39.439872 5583 catalog_manager.cc:5484] T 7f4b2166aae54490a36ced8d37caf7b1 P 12f3f54871994efeb833126452c61da9 reported cstate change: config changed from index 608 to 610, VOTER ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "12f3f54871994efeb833126452c61da9" committed_config { opid_index: 610 OBSOLETE_local: false peers { permanent_uuid: "12f3f54871994efeb833126452c61da9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 38041 } } peers { permanent_uuid: "abddb8eea0a34d4b9069ab7b89ff1e1a" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43495 } } }
I20240501 07:59:39.442947 5972 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f4b2166aae54490a36ced8d37caf7b1 with delete_type TABLET_DATA_TOMBSTONED (TS ec372f1e72b040e780e6bdd4d5409d33 not found in new config with opid_index 610) from {username='slave'} at 127.0.0.1:38970
I20240501 07:59:39.443306 10977 tablet_replica.cc:330] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33: stopping tablet replica
I20240501 07:59:39.443442 10977 raft_consensus.cc:2229] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:39.443545 10977 raft_consensus.cc:2258] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:39.445109 10977 ts_tablet_manager.cc:1875] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:39.447615 10977 ts_tablet_manager.cc:1888] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.609
I20240501 07:59:39.447716 10977 log.cc:1196] T 7f4b2166aae54490a36ced8d37caf7b1 P ec372f1e72b040e780e6bdd4d5409d33: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/7f4b2166aae54490a36ced8d37caf7b1
I20240501 07:59:39.448482 5575 catalog_manager.cc:4830] TS ec372f1e72b040e780e6bdd4d5409d33 (127.0.99.131:38531): tablet 7f4b2166aae54490a36ced8d37caf7b1 (table rebalance_test_table_2 [id=f3b4504f12d4447ba80f01cbb31b8c0a]) successfully deleted
I20240501 07:59:39.707574 6920 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.709138 9915 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.709738 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.719409 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.722291 5841 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:39.727594 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:39.728672 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:39.736683 5972 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:39.831070 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.833233 6920 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.833602 9915 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.835525 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:39.851289 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.857234 5972 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:39.862937 5841 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:39.875619 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:39.966986 7051 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.967633 5710 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:39.969282 7317 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:39.970075 5972 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:39.976493 9915 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.985469 7170 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:39.995077 5841 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:40.008591 6920 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6ba367f2ecb14c299239fdf04770cb81 | 127.0.99.190:40255 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:39909 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
12f3f54871994efeb833126452c61da9 | 127.0.99.132:38041 | HEALTHY | <none> | 1 | 0
46c782f15fb643acb2ef72105f1fb04a | 127.0.99.129:33797 | HEALTHY | <none> | 2 | 0
72a601327d9d447aa7957cde6c36e000 | 127.0.99.133:34267 | HEALTHY | <none> | 1 | 0
a3af86b0e9a04ecbb15d305c48d7acb4 | 127.0.99.135:45001 | HEALTHY | <none> | 2 | 0
abddb8eea0a34d4b9069ab7b89ff1e1a | 127.0.99.136:43495 | HEALTHY | <none> | 1 | 0
afc8c07c29354a04926027f57b8c6fab | 127.0.99.134:38017 | HEALTHY | <none> | 1 | 0
ec372f1e72b040e780e6bdd4d5409d33 | 127.0.99.131:38531 | HEALTHY | <none> | 4 | 0
f022e3632e9a428882a6cd16dc398723 | 127.0.99.130:42187 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:33797
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:42187
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:38531
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:38041
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:34267
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:38017
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:45001
local_ip_for_outbound_sockets | 127.0.99.136 | experimental | 127.0.99.136:43495
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:33797
server_dump_info_path | /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:42187
server_dump_info_path | /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:38531
server_dump_info_path | /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:38041
server_dump_info_path | /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:34267
server_dump_info_path | /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:38017
server_dump_info_path | /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:45001
server_dump_info_path | /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.99.136:43495
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:39909 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 07:59:40.033185 398 log_verifier.cc:126] Checking tablet 0776eb886275492285da625d69e77e6b
I20240501 07:59:40.126978 398 log_verifier.cc:177] Verified matching terms for 601 ops in tablet 0776eb886275492285da625d69e77e6b
I20240501 07:59:40.127177 398 log_verifier.cc:126] Checking tablet 114e9265cef4425799ad44e73e053b59
I20240501 07:59:40.217365 398 log_verifier.cc:177] Verified matching terms for 600 ops in tablet 114e9265cef4425799ad44e73e053b59
I20240501 07:59:40.217543 398 log_verifier.cc:126] Checking tablet 1e15c1939984497fb1795ef7502545ca
I20240501 07:59:40.303773 398 log_verifier.cc:177] Verified matching terms for 573 ops in tablet 1e15c1939984497fb1795ef7502545ca
I20240501 07:59:40.303946 398 log_verifier.cc:126] Checking tablet 26ad4c1ac47544f586d3f46f5dc9c77e
I20240501 07:59:40.406538 398 log_verifier.cc:177] Verified matching terms for 700 ops in tablet 26ad4c1ac47544f586d3f46f5dc9c77e
I20240501 07:59:40.406749 398 log_verifier.cc:126] Checking tablet 2a577537e9294fd5bd8507232d1d1a76
I20240501 07:59:40.509886 398 log_verifier.cc:177] Verified matching terms for 663 ops in tablet 2a577537e9294fd5bd8507232d1d1a76
I20240501 07:59:40.510082 398 log_verifier.cc:126] Checking tablet 3f187bcb4c494805a8d6f366732b695a
I20240501 07:59:40.606201 398 log_verifier.cc:177] Verified matching terms for 640 ops in tablet 3f187bcb4c494805a8d6f366732b695a
I20240501 07:59:40.606398 398 log_verifier.cc:126] Checking tablet 65e9dbbdb1fa459fabc258f9e6878f00
I20240501 07:59:40.704489 398 log_verifier.cc:177] Verified matching terms for 619 ops in tablet 65e9dbbdb1fa459fabc258f9e6878f00
I20240501 07:59:40.704682 398 log_verifier.cc:126] Checking tablet 662d37043f054aa1a562cde8d6c155a9
I20240501 07:59:40.799912 398 log_verifier.cc:177] Verified matching terms for 636 ops in tablet 662d37043f054aa1a562cde8d6c155a9
I20240501 07:59:40.800107 398 log_verifier.cc:126] Checking tablet 7f4b2166aae54490a36ced8d37caf7b1
I20240501 07:59:40.897701 398 log_verifier.cc:177] Verified matching terms for 610 ops in tablet 7f4b2166aae54490a36ced8d37caf7b1
I20240501 07:59:40.897915 398 log_verifier.cc:126] Checking tablet 877dfd1e30ee4e12b16674d96d57c48e
I20240501 07:59:41.005824 398 log_verifier.cc:177] Verified matching terms for 663 ops in tablet 877dfd1e30ee4e12b16674d96d57c48e
I20240501 07:59:41.006055 398 log_verifier.cc:126] Checking tablet a07321bfc4c1445a906ec7c1261693e9
I20240501 07:59:41.101740 398 log_verifier.cc:177] Verified matching terms for 579 ops in tablet a07321bfc4c1445a906ec7c1261693e9
I20240501 07:59:41.101959 398 log_verifier.cc:126] Checking tablet a6cb1009e4fb4fb0a8e7d55c2c24ff65
I20240501 07:59:41.198989 398 log_verifier.cc:177] Verified matching terms for 649 ops in tablet a6cb1009e4fb4fb0a8e7d55c2c24ff65
I20240501 07:59:41.199232 398 log_verifier.cc:126] Checking tablet ba935cdc94cf4eeab62cfee9e789e2eb
I20240501 07:59:41.208703 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet ba935cdc94cf4eeab62cfee9e789e2eb
I20240501 07:59:41.208833 398 log_verifier.cc:126] Checking tablet bd95a7eff75845c0b49790c9657ef1e0
I20240501 07:59:41.308212 398 log_verifier.cc:177] Verified matching terms for 669 ops in tablet bd95a7eff75845c0b49790c9657ef1e0
I20240501 07:59:41.308419 398 log_verifier.cc:126] Checking tablet d26c0973a2dc4c639bc3a616e28fbfef
I20240501 07:59:41.405624 398 log_verifier.cc:177] Verified matching terms for 632 ops in tablet d26c0973a2dc4c639bc3a616e28fbfef
I20240501 07:59:41.405819 398 log_verifier.cc:126] Checking tablet dad26af7f84e42208fd4461656117035
I20240501 07:59:41.499743 398 log_verifier.cc:177] Verified matching terms for 625 ops in tablet dad26af7f84e42208fd4461656117035
I20240501 07:59:41.501369 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 5649
I20240501 07:59:41.511220 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 5780
I20240501 07:59:41.517872 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 5911
I20240501 07:59:41.524720 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 6858
I20240501 07:59:41.530381 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 6989
I20240501 07:59:41.536120 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 7120
I20240501 07:59:41.541893 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 7252
I20240501 07:59:41.547864 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 9853
I20240501 07:59:41.553331 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 5558
2024-05-01T07:59:41Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/2 (39610 ms)
[----------] 1 test from RebalanceParamTest (39610 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240501 07:59:41.602923 398 test_util.cc:268] Using random seed: -562960282
I20240501 07:59:41.603686 398 ts_itest-base.cc:112] Starting cluster with:
I20240501 07:59:41.603725 398 ts_itest-base.cc:113] --------------
I20240501 07:59:41.603734 398 ts_itest-base.cc:114] 8 tablet servers
I20240501 07:59:41.603744 398 ts_itest-base.cc:115] 3 replicas per TS
I20240501 07:59:41.603752 398 ts_itest-base.cc:116] --------------
2024-05-01T07:59:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T07:59:41Z Disabled control of system clock
I20240501 07:59:41.612323 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:41001
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:41001
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 07:59:41.722676 11097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:41.722934 11097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:41.722978 11097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:41.726202 11097 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:41.726305 11097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 07:59:41.726329 11097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:41.726365 11097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 07:59:41.726397 11097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 07:59:41.730391 11097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:41001
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:41001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.11097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:41.731388 11097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:41.732530 11097 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:41.740330 11103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.740813 11105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:41.741271 11102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:41.741838 11097 server_base.cc:1034] running on GCE node
I20240501 07:59:41.742406 11097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:41.743445 11097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:41.744606 11097 hybrid_clock.cc:648] HybridClock initialized: now 1714550381744593 us; error 29 us; skew 500 ppm
I20240501 07:59:41.746780 11097 webserver.cc:460] Webserver started at http://127.0.99.190:43721/ using document root <none> and password file <none>
I20240501 07:59:41.747381 11097 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:41.747443 11097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:41.747640 11097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:41.749404 11097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2ed840e35a9f4afe9077d15510856d4e"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-2srn"
I20240501 07:59:41.749903 11097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2ed840e35a9f4afe9077d15510856d4e"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-2srn"
I20240501 07:59:41.753528 11097 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240501 07:59:41.755997 11111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:41.757064 11097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:41.757211 11097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2ed840e35a9f4afe9077d15510856d4e"
format_stamp: "Formatted at 2024-05-01 07:59:41 on dist-test-slave-2srn"
I20240501 07:59:41.757316 11097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:41.775452 11097 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:41.776149 11097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:41.776327 11097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:41.782917 11097 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:41001
I20240501 07:59:41.782919 11163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:41001 every 8 connection(s)
I20240501 07:59:41.784008 11097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 07:59:41.786896 11164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:41.788457 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 11097
I20240501 07:59:41.788623 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 07:59:41.793553 11164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e: Bootstrap starting.
I20240501 07:59:41.795984 11164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:41.796792 11164 log.cc:824] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:41.798661 11164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e: No bootstrap required, opened a new log
I20240501 07:59:41.801450 11164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed840e35a9f4afe9077d15510856d4e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41001 } }
I20240501 07:59:41.801679 11164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:41.801717 11164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ed840e35a9f4afe9077d15510856d4e, State: Initialized, Role: FOLLOWER
I20240501 07:59:41.802156 11164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed840e35a9f4afe9077d15510856d4e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41001 } }
I20240501 07:59:41.802276 11164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:59:41.802337 11164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:59:41.802471 11164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:41.803332 11164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed840e35a9f4afe9077d15510856d4e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41001 } }
I20240501 07:59:41.803735 11164 leader_election.cc:302] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ed840e35a9f4afe9077d15510856d4e; no voters:
I20240501 07:59:41.804070 11164 leader_election.cc:288] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:59:41.804222 11169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:41.804816 11169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e [term 1 LEADER]: Becoming Leader. State: Replica: 2ed840e35a9f4afe9077d15510856d4e, State: Running, Role: LEADER
I20240501 07:59:41.805243 11164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:59:41.805224 11169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed840e35a9f4afe9077d15510856d4e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41001 } }
I20240501 07:59:41.806731 11171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ed840e35a9f4afe9077d15510856d4e. Latest consensus state: current_term: 1 leader_uuid: "2ed840e35a9f4afe9077d15510856d4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed840e35a9f4afe9077d15510856d4e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41001 } } }
I20240501 07:59:41.806857 11171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:41.807150 11170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ed840e35a9f4afe9077d15510856d4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed840e35a9f4afe9077d15510856d4e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41001 } } }
I20240501 07:59:41.807204 11180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:59:41.807235 11170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e [sys.catalog]: This master's current role is: LEADER
I20240501 07:59:41.810145 11180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:59:41.816076 11180 catalog_manager.cc:1347] Generated new cluster ID: c53b2e5c9ac64922b877cfc970f78f20
I20240501 07:59:41.816174 11180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:59:41.846911 11180 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:59:41.848235 11180 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:59:41.870885 11180 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2ed840e35a9f4afe9077d15510856d4e: Generated new TSK 0
I20240501 07:59:41.871868 11180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:59:41.886351 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41001
--builtin_ntp_servers=127.0.99.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 07:59:41.998665 11188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:41.999023 11188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:41.999075 11188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:42.002502 11188 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:42.002588 11188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:42.002661 11188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240501 07:59:42.006736 11188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41001
--never_fsync=true
--heap_profile_path=/tmp/kudu.11188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:42.007732 11188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:42.008889 11188 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:42.017802 11193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:42.018239 11196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:42.018687 11194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:42.019306 11188 server_base.cc:1034] running on GCE node
I20240501 07:59:42.019749 11188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:42.020378 11188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:42.021567 11188 hybrid_clock.cc:648] HybridClock initialized: now 1714550382021547 us; error 33 us; skew 500 ppm
I20240501 07:59:42.023741 11188 webserver.cc:460] Webserver started at http://127.0.99.129:41817/ using document root <none> and password file <none>
I20240501 07:59:42.024340 11188 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:42.024406 11188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:42.024587 11188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:42.026364 11188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:42.026902 11188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:42.030598 11188 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:59:42.033244 11202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:42.034503 11188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:42.034636 11188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:42.034739 11188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:42.057116 11188 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:42.057811 11188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:42.057976 11188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:42.058552 11188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:42.059504 11188 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:42.059552 11188 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:42.059597 11188 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:42.059626 11188 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:42.068725 11188 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:46085
I20240501 07:59:42.068754 11315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:46085 every 8 connection(s)
I20240501 07:59:42.069808 11188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 07:59:42.072649 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 11188
I20240501 07:59:42.072779 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 07:59:42.075160 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41001
--builtin_ntp_servers=127.0.99.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:42.079838 11316 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41001
I20240501 07:59:42.080152 11316 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:42.080809 11316 heartbeater.cc:511] Master 127.0.99.190:41001 requested a full tablet report, sending...
I20240501 07:59:42.082515 11128 ts_manager.cc:194] Registered new tserver with Master: e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 07:59:42.083719 11128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:38101
W20240501 07:59:42.187616 11319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:42.187883 11319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:42.187927 11319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:42.191256 11319 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:42.191344 11319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:42.191434 11319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240501 07:59:42.195590 11319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41001
--never_fsync=true
--heap_profile_path=/tmp/kudu.11319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:42.196621 11319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:42.197858 11319 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:42.207059 11327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:42.207470 11324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:42.207985 11325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:42.208489 11319 server_base.cc:1034] running on GCE node
I20240501 07:59:42.208958 11319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:42.209605 11319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:42.210752 11319 hybrid_clock.cc:648] HybridClock initialized: now 1714550382210736 us; error 35 us; skew 500 ppm
I20240501 07:59:42.212894 11319 webserver.cc:460] Webserver started at http://127.0.99.130:44065/ using document root <none> and password file <none>
I20240501 07:59:42.213486 11319 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:42.213547 11319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:42.213752 11319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:42.215595 11319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "655ebda1fb2c49c4af7087a760529835"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:42.216118 11319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "655ebda1fb2c49c4af7087a760529835"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:42.219763 11319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:59:42.222186 11333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:42.223331 11319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:42.223449 11319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "655ebda1fb2c49c4af7087a760529835"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:42.223558 11319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:42.267952 11319 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:42.268646 11319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:42.268824 11319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:42.269400 11319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:42.270339 11319 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:42.270390 11319 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:42.270459 11319 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:42.270479 11319 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:42.279110 11319 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:41575
I20240501 07:59:42.279160 11446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:41575 every 8 connection(s)
I20240501 07:59:42.280167 11319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 07:59:42.281702 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 11319
I20240501 07:59:42.281805 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 07:59:42.284229 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41001
--builtin_ntp_servers=127.0.99.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:42.290946 11447 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41001
I20240501 07:59:42.291218 11447 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:42.291740 11447 heartbeater.cc:511] Master 127.0.99.190:41001 requested a full tablet report, sending...
I20240501 07:59:42.292837 11128 ts_manager.cc:194] Registered new tserver with Master: 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:42.293417 11128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:45123
W20240501 07:59:42.396404 11450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:42.396695 11450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:42.396757 11450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:42.399986 11450 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:42.400085 11450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:42.400180 11450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240501 07:59:42.404307 11450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41001
--never_fsync=true
--heap_profile_path=/tmp/kudu.11450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:42.405272 11450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:42.406492 11450 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:42.415129 11455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:42.415593 11458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:42.416078 11456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:42.416662 11450 server_base.cc:1034] running on GCE node
I20240501 07:59:42.417131 11450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:42.417768 11450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:42.418915 11450 hybrid_clock.cc:648] HybridClock initialized: now 1714550382418890 us; error 44 us; skew 500 ppm
I20240501 07:59:42.421065 11450 webserver.cc:460] Webserver started at http://127.0.99.131:38227/ using document root <none> and password file <none>
I20240501 07:59:42.421634 11450 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:42.421694 11450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:42.421851 11450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:42.423604 11450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "22ec12774a4047a8b859743dc73f8ff6"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:42.424050 11450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "22ec12774a4047a8b859743dc73f8ff6"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:42.427559 11450 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:59:42.429970 11464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:42.431118 11450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:42.431257 11450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "22ec12774a4047a8b859743dc73f8ff6"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:42.431370 11450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:42.454605 11450 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:42.455291 11450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:42.455457 11450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:42.455999 11450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:42.456935 11450 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:42.456983 11450 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:42.457026 11450 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:42.457053 11450 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:42.465615 11450 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:37957
I20240501 07:59:42.465641 11577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:37957 every 8 connection(s)
I20240501 07:59:42.466670 11450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 07:59:42.470997 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 11450
I20240501 07:59:42.471129 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 07:59:42.473366 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41001
--builtin_ntp_servers=127.0.99.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:42.476534 11578 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41001
I20240501 07:59:42.476785 11578 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:42.477299 11578 heartbeater.cc:511] Master 127.0.99.190:41001 requested a full tablet report, sending...
I20240501 07:59:42.478279 11128 ts_manager.cc:194] Registered new tserver with Master: 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:42.478878 11128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:52863
W20240501 07:59:42.584853 11581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:42.585130 11581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:42.585192 11581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:42.588471 11581 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:42.588572 11581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:42.588670 11581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:59:42.592510 11581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41001
--never_fsync=true
--heap_profile_path=/tmp/kudu.11581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:42.593488 11581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:42.594669 11581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:42.603433 11586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:42.603808 11589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:42.604333 11587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:42.604907 11581 server_base.cc:1034] running on GCE node
I20240501 07:59:42.605365 11581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:42.606009 11581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:42.607200 11581 hybrid_clock.cc:648] HybridClock initialized: now 1714550382607180 us; error 33 us; skew 500 ppm
I20240501 07:59:42.609335 11581 webserver.cc:460] Webserver started at http://127.0.99.132:33729/ using document root <none> and password file <none>
I20240501 07:59:42.609925 11581 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:42.609984 11581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:42.610177 11581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:42.611999 11581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "603919bb02364c04b6c530245b8b02f2"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:42.612517 11581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "603919bb02364c04b6c530245b8b02f2"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:42.616150 11581 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240501 07:59:42.618546 11595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:42.619639 11581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:42.619762 11581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "603919bb02364c04b6c530245b8b02f2"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:42.619887 11581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:42.648814 11581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:42.649490 11581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:42.649649 11581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:42.650200 11581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:42.651160 11581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:42.651211 11581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:42.651252 11581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:42.651273 11581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:42.659529 11581 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:44403
I20240501 07:59:42.659554 11708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:44403 every 8 connection(s)
I20240501 07:59:42.660494 11581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:59:42.669111 11709 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41001
I20240501 07:59:42.669358 11709 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:42.669853 11709 heartbeater.cc:511] Master 127.0.99.190:41001 requested a full tablet report, sending...
I20240501 07:59:42.669859 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 11581
I20240501 07:59:42.669957 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 07:59:42.670936 11128 ts_manager.cc:194] Registered new tserver with Master: 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:42.671548 11128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:60571
I20240501 07:59:42.672224 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41001
--builtin_ntp_servers=127.0.99.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 07:59:42.780679 11712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:42.780946 11712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:42.780989 11712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:42.784323 11712 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:42.784430 11712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:42.784528 11712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:59:42.788489 11712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41001
--never_fsync=true
--heap_profile_path=/tmp/kudu.11712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:42.789487 11712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:42.790797 11712 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:42.799744 11718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:42.800119 11717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:42.800642 11720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:42.801293 11712 server_base.cc:1034] running on GCE node
I20240501 07:59:42.801724 11712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:42.802294 11712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:42.803478 11712 hybrid_clock.cc:648] HybridClock initialized: now 1714550382803452 us; error 41 us; skew 500 ppm
I20240501 07:59:42.805747 11712 webserver.cc:460] Webserver started at http://127.0.99.133:44695/ using document root <none> and password file <none>
I20240501 07:59:42.806376 11712 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:42.806459 11712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:42.806665 11712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:42.808580 11712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "60e8857abe7c41038af46ebb5a095a90"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:42.809072 11712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "60e8857abe7c41038af46ebb5a095a90"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:42.812731 11712 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240501 07:59:42.815191 11726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:42.816278 11712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:42.816391 11712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "60e8857abe7c41038af46ebb5a095a90"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:42.816496 11712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:42.833081 11712 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:42.833748 11712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:42.833912 11712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:42.834520 11712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:42.835536 11712 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:42.835600 11712 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:42.835646 11712 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:42.835675 11712 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:42.844293 11839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:42805 every 8 connection(s)
I20240501 07:59:42.844272 11712 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:42805
I20240501 07:59:42.845364 11712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:59:42.847810 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 11712
I20240501 07:59:42.847937 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 07:59:42.850215 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41001
--builtin_ntp_servers=127.0.99.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:42.854820 11840 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41001
I20240501 07:59:42.855058 11840 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:42.855562 11840 heartbeater.cc:511] Master 127.0.99.190:41001 requested a full tablet report, sending...
I20240501 07:59:42.856662 11128 ts_manager.cc:194] Registered new tserver with Master: 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
I20240501 07:59:42.857405 11128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:59391
W20240501 07:59:42.963783 11843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:42.964046 11843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:42.964090 11843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:42.967386 11843 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:42.967474 11843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:42.967545 11843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 07:59:42.971380 11843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41001
--never_fsync=true
--heap_profile_path=/tmp/kudu.11843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:42.972326 11843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:42.973405 11843 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:42.981802 11851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:42.982264 11848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:42.982803 11849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:42.983331 11843 server_base.cc:1034] running on GCE node
I20240501 07:59:42.983759 11843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:42.984352 11843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:42.985512 11843 hybrid_clock.cc:648] HybridClock initialized: now 1714550382985497 us; error 33 us; skew 500 ppm
I20240501 07:59:42.987602 11843 webserver.cc:460] Webserver started at http://127.0.99.134:42815/ using document root <none> and password file <none>
I20240501 07:59:42.988189 11843 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:42.988255 11843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:42.988436 11843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:42.990159 11843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "53d1d2c0767b4f6195b69758a3dd6cf9"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:42.990679 11843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "53d1d2c0767b4f6195b69758a3dd6cf9"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:42.994196 11843 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240501 07:59:42.996695 11857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:42.997825 11843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:42.997953 11843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "53d1d2c0767b4f6195b69758a3dd6cf9"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:42.998057 11843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:43.021103 11843 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:43.021750 11843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:43.021907 11843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:43.022472 11843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:43.023444 11843 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:43.023494 11843 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:43.023548 11843 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:43.023573 11843 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:43.032083 11843 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:36771
I20240501 07:59:43.032101 11970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:36771 every 8 connection(s)
I20240501 07:59:43.033090 11843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 07:59:43.036432 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 11843
I20240501 07:59:43.036571 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 07:59:43.038647 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41001
--builtin_ntp_servers=127.0.99.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:43.042620 11971 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41001
I20240501 07:59:43.042878 11971 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:43.043377 11971 heartbeater.cc:511] Master 127.0.99.190:41001 requested a full tablet report, sending...
I20240501 07:59:43.044492 11128 ts_manager.cc:194] Registered new tserver with Master: 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771)
I20240501 07:59:43.045292 11128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:59913
I20240501 07:59:43.086833 11316 heartbeater.cc:503] Master 127.0.99.190:41001 was elected leader, sending a full tablet report...
W20240501 07:59:43.150369 11974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:43.150651 11974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:43.150701 11974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:43.153887 11974 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:43.153976 11974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:43.154054 11974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 07:59:43.157971 11974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41001
--never_fsync=true
--heap_profile_path=/tmp/kudu.11974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:43.159005 11974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:43.160140 11974 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:43.168907 11982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:43.169359 11980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:43.169965 11979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:43.170635 11974 server_base.cc:1034] running on GCE node
I20240501 07:59:43.171047 11974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:43.171643 11974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:43.172793 11974 hybrid_clock.cc:648] HybridClock initialized: now 1714550383172780 us; error 28 us; skew 500 ppm
I20240501 07:59:43.174842 11974 webserver.cc:460] Webserver started at http://127.0.99.135:37353/ using document root <none> and password file <none>
I20240501 07:59:43.175410 11974 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:43.175465 11974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:43.175648 11974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:43.177366 11974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "e5c03261d31c4d9a91077a17e1cd0336"
format_stamp: "Formatted at 2024-05-01 07:59:43 on dist-test-slave-2srn"
I20240501 07:59:43.177852 11974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "e5c03261d31c4d9a91077a17e1cd0336"
format_stamp: "Formatted at 2024-05-01 07:59:43 on dist-test-slave-2srn"
I20240501 07:59:43.181203 11974 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240501 07:59:43.183445 11988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:43.184500 11974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:43.184623 11974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "e5c03261d31c4d9a91077a17e1cd0336"
format_stamp: "Formatted at 2024-05-01 07:59:43 on dist-test-slave-2srn"
I20240501 07:59:43.184720 11974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:43.213225 11974 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:43.213825 11974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:43.213973 11974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:43.214555 11974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:43.215458 11974 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:43.215507 11974 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:43.215561 11974 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:43.215585 11974 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:43.223989 11974 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:43995
I20240501 07:59:43.224002 12101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:43995 every 8 connection(s)
I20240501 07:59:43.224975 11974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 07:59:43.233583 12102 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41001
I20240501 07:59:43.233831 12102 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:43.234328 12102 heartbeater.cc:511] Master 127.0.99.190:41001 requested a full tablet report, sending...
I20240501 07:59:43.235257 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 11974
I20240501 07:59:43.235368 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 07:59:43.235428 11128 ts_manager.cc:194] Registered new tserver with Master: e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
I20240501 07:59:43.236027 11128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:59609
I20240501 07:59:43.237371 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41001
--builtin_ntp_servers=127.0.99.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:43.295727 11447 heartbeater.cc:503] Master 127.0.99.190:41001 was elected leader, sending a full tablet report...
W20240501 07:59:43.347242 12105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:43.347494 12105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:43.347539 12105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:43.350737 12105 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:43.350822 12105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:43.350896 12105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240501 07:59:43.355803 12105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41001
--never_fsync=true
--heap_profile_path=/tmp/kudu.12105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:43.357085 12105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:43.358656 12105 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:43.368536 12110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:43.369050 12111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:43.369655 12113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:43.370220 12105 server_base.cc:1034] running on GCE node
I20240501 07:59:43.370715 12105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:43.371316 12105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:43.372467 12105 hybrid_clock.cc:648] HybridClock initialized: now 1714550383372457 us; error 35 us; skew 500 ppm
I20240501 07:59:43.374650 12105 webserver.cc:460] Webserver started at http://127.0.99.136:36501/ using document root <none> and password file <none>
I20240501 07:59:43.375244 12105 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:43.375306 12105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:43.375483 12105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:59:43.377242 12105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "ca177138f2c44d2eafa1646b795f7608"
format_stamp: "Formatted at 2024-05-01 07:59:43 on dist-test-slave-2srn"
I20240501 07:59:43.377735 12105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "ca177138f2c44d2eafa1646b795f7608"
format_stamp: "Formatted at 2024-05-01 07:59:43 on dist-test-slave-2srn"
I20240501 07:59:43.381474 12105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20240501 07:59:43.383988 12119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:43.385159 12105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:43.385309 12105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "ca177138f2c44d2eafa1646b795f7608"
format_stamp: "Formatted at 2024-05-01 07:59:43 on dist-test-slave-2srn"
I20240501 07:59:43.385419 12105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:43.402607 12105 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:43.403280 12105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:43.403429 12105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:43.403957 12105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:43.404877 12105 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:43.404927 12105 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:43.404979 12105 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:43.405001 12105 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:43.413298 12105 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:34503
I20240501 07:59:43.413372 12232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:34503 every 8 connection(s)
I20240501 07:59:43.414273 12105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 07:59:43.422963 12233 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41001
I20240501 07:59:43.423249 12233 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:43.423605 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 12105
I20240501 07:59:43.423725 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 07:59:43.423738 12233 heartbeater.cc:511] Master 127.0.99.190:41001 requested a full tablet report, sending...
I20240501 07:59:43.424805 11128 ts_manager.cc:194] Registered new tserver with Master: ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503)
I20240501 07:59:43.425487 11128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:43065
I20240501 07:59:43.425987 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 07:59:43.435274 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 07:59:43.435361 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 11712
I20240501 07:59:43.440326 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 11843
I20240501 07:59:43.445300 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 11974
I20240501 07:59:43.450671 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu with pid 12105
I20240501 07:59:43.480866 11578 heartbeater.cc:503] Master 127.0.99.190:41001 was elected leader, sending a full tablet report...
I20240501 07:59:43.673736 11709 heartbeater.cc:503] Master 127.0.99.190:41001 was elected leader, sending a full tablet report...
I20240501 07:59:47.207160 11128 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49370:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:47.227447 11381 tablet_service.cc:1467] Processing CreateTablet for tablet c95c4f3ef08f44d1b69fb7b98426f5af (DEFAULT_TABLE table=rebalance_test_table_0 [id=24b4f6e8cc8c44f6961591cac7beacb4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.228528 11381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95c4f3ef08f44d1b69fb7b98426f5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.228840 11512 tablet_service.cc:1467] Processing CreateTablet for tablet c95c4f3ef08f44d1b69fb7b98426f5af (DEFAULT_TABLE table=rebalance_test_table_0 [id=24b4f6e8cc8c44f6961591cac7beacb4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.229732 11512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95c4f3ef08f44d1b69fb7b98426f5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.229547 11249 tablet_service.cc:1467] Processing CreateTablet for tablet a1ebf29682024645978b3caa2d4988f3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=24b4f6e8cc8c44f6961591cac7beacb4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.230010 11248 tablet_service.cc:1467] Processing CreateTablet for tablet a24be92aec7544089c7818f2802b9c85 (DEFAULT_TABLE table=rebalance_test_table_0 [id=24b4f6e8cc8c44f6961591cac7beacb4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.229553 11250 tablet_service.cc:1467] Processing CreateTablet for tablet c95c4f3ef08f44d1b69fb7b98426f5af (DEFAULT_TABLE table=rebalance_test_table_0 [id=24b4f6e8cc8c44f6961591cac7beacb4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.230407 11249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1ebf29682024645978b3caa2d4988f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.230706 11250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95c4f3ef08f44d1b69fb7b98426f5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.230736 11511 tablet_service.cc:1467] Processing CreateTablet for tablet a24be92aec7544089c7818f2802b9c85 (DEFAULT_TABLE table=rebalance_test_table_0 [id=24b4f6e8cc8c44f6961591cac7beacb4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.230978 11511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a24be92aec7544089c7818f2802b9c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.231451 11248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a24be92aec7544089c7818f2802b9c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.234916 11380 tablet_service.cc:1467] Processing CreateTablet for tablet a1ebf29682024645978b3caa2d4988f3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=24b4f6e8cc8c44f6961591cac7beacb4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.235258 11380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1ebf29682024645978b3caa2d4988f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.235383 11643 tablet_service.cc:1467] Processing CreateTablet for tablet a1ebf29682024645978b3caa2d4988f3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=24b4f6e8cc8c44f6961591cac7beacb4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.236379 11643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1ebf29682024645978b3caa2d4988f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.238330 11642 tablet_service.cc:1467] Processing CreateTablet for tablet a24be92aec7544089c7818f2802b9c85 (DEFAULT_TABLE table=rebalance_test_table_0 [id=24b4f6e8cc8c44f6961591cac7beacb4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.238643 11642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a24be92aec7544089c7818f2802b9c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.240485 12254 tablet_bootstrap.cc:492] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:47.241042 12253 tablet_bootstrap.cc:492] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:47.242671 12255 tablet_bootstrap.cc:492] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:47.242738 12254 tablet_bootstrap.cc:654] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.242946 12253 tablet_bootstrap.cc:654] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.243605 12254 log.cc:824] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:47.243639 12253 log.cc:824] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:47.244732 12255 tablet_bootstrap.cc:654] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.245064 12256 tablet_bootstrap.cc:492] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:47.245684 12255 log.cc:824] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:47.246495 12254 tablet_bootstrap.cc:492] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:47.246758 12254 ts_tablet_manager.cc:1367] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 07:59:47.246822 12253 tablet_bootstrap.cc:492] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:47.247001 12253 ts_tablet_manager.cc:1367] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 07:59:47.247133 12256 tablet_bootstrap.cc:654] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.247288 12255 tablet_bootstrap.cc:492] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:47.247452 12255 ts_tablet_manager.cc:1367] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:59:47.247924 12256 log.cc:824] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:47.249809 12256 tablet_bootstrap.cc:492] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:47.249991 12256 ts_tablet_manager.cc:1367] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240501 07:59:47.250524 12253 raft_consensus.cc:348] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.250766 12253 raft_consensus.cc:374] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.250816 12253 raft_consensus.cc:729] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.250895 12255 raft_consensus.cc:348] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:47.251063 12255 raft_consensus.cc:374] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.251112 12255 raft_consensus.cc:729] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.251338 12253 consensus_queue.cc:260] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.251514 12255 consensus_queue.cc:260] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:47.251961 12254 raft_consensus.cc:348] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.252178 12254 raft_consensus.cc:374] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.252226 12254 raft_consensus.cc:729] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.252318 12255 ts_tablet_manager.cc:1398] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:59:47.252318 12253 ts_tablet_manager.cc:1398] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 07:59:47.252717 12253 tablet_bootstrap.cc:492] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:47.253099 12254 consensus_queue.cc:260] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.253818 12253 tablet_bootstrap.cc:654] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.253932 12256 raft_consensus.cc:348] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:47.254159 12256 raft_consensus.cc:374] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.254220 12256 raft_consensus.cc:729] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.254151 12254 ts_tablet_manager.cc:1398] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 07:59:47.255043 12255 tablet_bootstrap.cc:492] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:47.255467 12256 consensus_queue.cc:260] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:47.256033 12254 tablet_bootstrap.cc:492] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:47.257009 12256 ts_tablet_manager.cc:1398] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 07:59:47.257362 12256 tablet_bootstrap.cc:492] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:47.257695 12254 tablet_bootstrap.cc:654] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.258309 12256 tablet_bootstrap.cc:654] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.258852 12255 tablet_bootstrap.cc:654] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.260217 12253 tablet_bootstrap.cc:492] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:47.260377 12253 ts_tablet_manager.cc:1367] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:59:47.260393 12256 tablet_bootstrap.cc:492] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:47.260491 12256 ts_tablet_manager.cc:1367] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:47.261075 12256 raft_consensus.cc:348] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.261065 12253 raft_consensus.cc:348] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:47.261180 12256 raft_consensus.cc:374] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.261193 12253 raft_consensus.cc:374] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.261222 12256 raft_consensus.cc:729] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.261299 12253 raft_consensus.cc:729] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.261458 12256 consensus_queue.cc:260] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.261513 12253 consensus_queue.cc:260] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:47.261787 12256 ts_tablet_manager.cc:1398] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:47.261811 12253 ts_tablet_manager.cc:1398] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:47.261955 12253 tablet_bootstrap.cc:492] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:47.261976 12254 tablet_bootstrap.cc:492] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:47.262082 12254 ts_tablet_manager.cc:1367] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:47.262727 12254 raft_consensus.cc:348] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:47.262859 12254 raft_consensus.cc:374] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.262892 12254 raft_consensus.cc:729] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.263051 12253 tablet_bootstrap.cc:654] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.263011 12254 consensus_queue.cc:260] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:47.263367 12254 ts_tablet_manager.cc:1398] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:47.264432 12253 tablet_bootstrap.cc:492] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:47.264577 12253 ts_tablet_manager.cc:1367] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:59:47.264845 12255 tablet_bootstrap.cc:492] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:47.264947 12255 ts_tablet_manager.cc:1367] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 07:59:47.265092 12253 raft_consensus.cc:348] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:47.265241 12253 raft_consensus.cc:374] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.265312 12253 raft_consensus.cc:729] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.265444 12253 consensus_queue.cc:260] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:47.265556 12255 raft_consensus.cc:348] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:47.265671 12255 raft_consensus.cc:374] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.265733 12255 raft_consensus.cc:729] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.265854 12253 ts_tablet_manager.cc:1398] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:47.265885 12255 consensus_queue.cc:260] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:47.266207 12255 ts_tablet_manager.cc:1398] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 07:59:47.284714 11448 tablet.cc:2369] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:47.289009 12261 raft_consensus.cc:482] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:47.289228 12261 raft_consensus.cc:504] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:47.290366 12261 leader_election.cc:288] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:47.294312 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24be92aec7544089c7818f2802b9c85" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6" is_pre_election: true
I20240501 07:59:47.294405 11663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24be92aec7544089c7818f2802b9c85" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2" is_pre_election: true
I20240501 07:59:47.294664 11532 raft_consensus.cc:2454] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 0.
I20240501 07:59:47.294701 11663 raft_consensus.cc:2454] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 0.
I20240501 07:59:47.295122 11203 leader_election.cc:302] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:47.295405 12261 raft_consensus.cc:2781] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:47.295477 12261 raft_consensus.cc:482] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:47.295506 12261 raft_consensus.cc:3037] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.296388 12261 raft_consensus.cc:504] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:47.296780 12261 leader_election.cc:288] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 election: Requested vote from peers 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:47.297173 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24be92aec7544089c7818f2802b9c85" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
I20240501 07:59:47.297313 11663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24be92aec7544089c7818f2802b9c85" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 07:59:47.297359 11532 raft_consensus.cc:3037] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.297462 11663 raft_consensus.cc:3037] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.298291 11532 raft_consensus.cc:2454] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 1.
I20240501 07:59:47.298323 11663 raft_consensus.cc:2454] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 1.
I20240501 07:59:47.298736 11203 leader_election.cc:302] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:47.298969 12261 raft_consensus.cc:2781] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:47.299213 12261 raft_consensus.cc:686] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Becoming Leader. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Running, Role: LEADER
I20240501 07:59:47.299568 12261 consensus_queue.cc:237] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:47.302685 11127 catalog_manager.cc:5484] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
W20240501 07:59:47.324966 11317 tablet.cc:2369] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:47.346132 12263 raft_consensus.cc:482] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:47.346345 12263 raft_consensus.cc:504] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.347537 12263 leader_election.cc:288] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:47.351184 11663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ebf29682024645978b3caa2d4988f3" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2" is_pre_election: true
I20240501 07:59:47.351262 11270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ebf29682024645978b3caa2d4988f3" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" is_pre_election: true
I20240501 07:59:47.351348 11663 raft_consensus.cc:2454] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 0.
I20240501 07:59:47.351490 11270 raft_consensus.cc:2454] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 0.
I20240501 07:59:47.351732 11334 leader_election.cc:302] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, 655ebda1fb2c49c4af7087a760529835; no voters:
I20240501 07:59:47.352023 12263 raft_consensus.cc:2781] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:47.352094 12263 raft_consensus.cc:482] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:47.352125 12263 raft_consensus.cc:3037] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.353137 12263 raft_consensus.cc:504] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.353567 12263 leader_election.cc:288] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 election: Requested vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:47.354112 11663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ebf29682024645978b3caa2d4988f3" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 07:59:47.354251 11663 raft_consensus.cc:3037] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.354275 11270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1ebf29682024645978b3caa2d4988f3" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 07:59:47.354436 11270 raft_consensus.cc:3037] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.355263 11663 raft_consensus.cc:2454] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 1.
I20240501 07:59:47.355512 11270 raft_consensus.cc:2454] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 1.
I20240501 07:59:47.355816 11334 leader_election.cc:302] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, 655ebda1fb2c49c4af7087a760529835; no voters:
I20240501 07:59:47.356046 12263 raft_consensus.cc:2781] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:47.356294 12263 raft_consensus.cc:686] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Becoming Leader. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Running, Role: LEADER
I20240501 07:59:47.356925 12263 consensus_queue.cc:237] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.360755 11127 catalog_manager.cc:5484] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: term changed from 0 to 1, leader changed from <none> to 655ebda1fb2c49c4af7087a760529835 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
W20240501 07:59:47.414930 11710 tablet.cc:2369] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:59:47.471729 11579 tablet.cc:2369] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:47.477391 12261 raft_consensus.cc:482] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:47.477530 12261 raft_consensus.cc:504] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:47.478051 12261 leader_election.cc:288] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:47.478564 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95c4f3ef08f44d1b69fb7b98426f5af" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6" is_pre_election: true
I20240501 07:59:47.478729 11532 raft_consensus.cc:2454] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 0.
I20240501 07:59:47.479049 11205 leader_election.cc:302] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:47.479251 12261 raft_consensus.cc:2781] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:47.479314 12261 raft_consensus.cc:482] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:47.479354 12261 raft_consensus.cc:3037] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.480149 12261 raft_consensus.cc:504] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:47.480567 12261 leader_election.cc:288] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 election: Requested vote from peers 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:47.480965 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95c4f3ef08f44d1b69fb7b98426f5af" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
I20240501 07:59:47.481086 11532 raft_consensus.cc:3037] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.482091 11532 raft_consensus.cc:2454] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 1.
I20240501 07:59:47.482232 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95c4f3ef08f44d1b69fb7b98426f5af" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835" is_pre_election: true
I20240501 07:59:47.482228 11400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95c4f3ef08f44d1b69fb7b98426f5af" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 07:59:47.482450 11205 leader_election.cc:302] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:47.482498 11400 raft_consensus.cc:3037] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.482635 12261 raft_consensus.cc:2781] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:47.482836 12261 raft_consensus.cc:686] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Becoming Leader. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Running, Role: LEADER
I20240501 07:59:47.482992 12261 consensus_queue.cc:237] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:47.483450 11400 raft_consensus.cc:2454] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 1.
I20240501 07:59:47.484644 11127 catalog_manager.cc:5484] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
W20240501 07:59:47.625155 12279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:47.625451 12279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:47.628813 12279 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 07:59:47.717223 11663 raft_consensus.cc:1261] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:47.717267 11532 raft_consensus.cc:1261] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:47.717445 11270 raft_consensus.cc:1261] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:47.717790 11531 raft_consensus.cc:1261] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:47.717969 12263 consensus_queue.cc:1035] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:47.718120 11662 raft_consensus.cc:1261] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:47.718520 12263 consensus_queue.cc:1035] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:47.718827 12261 consensus_queue.cc:1035] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:47.720271 11400 raft_consensus.cc:1261] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:47.724654 12261 consensus_queue.cc:1035] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:47.725036 12261 consensus_queue.cc:1035] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:47.725575 12310 mvcc.cc:205] Tried to move back new op lower bound from 7022798388079140864 to 7022798386617225216. Current Snapshot: MvccSnapshot[applied={T|T < 7022798388079140864}]
I20240501 07:59:47.727061 12308 mvcc.cc:205] Tried to move back new op lower bound from 7022798388079140864 to 7022798386617225216. Current Snapshot: MvccSnapshot[applied={T|T < 7022798388079140864}]
I20240501 07:59:47.733208 12312 consensus_queue.cc:1035] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:47.741876 12318 mvcc.cc:205] Tried to move back new op lower bound from 7022798388079140864 to 7022798386617225216. Current Snapshot: MvccSnapshot[applied={T|T < 7022798388079140864}]
I20240501 07:59:47.759727 12326 mvcc.cc:205] Tried to move back new op lower bound from 7022798388115775488 to 7022798386382000128. Current Snapshot: MvccSnapshot[applied={T|T < 7022798388079755264}]
INSERT report
rows total: 2000
time total: 118.692 ms
time per row: 0.0593459 ms
I20240501 07:59:47.794020 11124 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49370:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:47.800698 11512 tablet_service.cc:1467] Processing CreateTablet for tablet 93602521de174997bda6702d25aafce1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=98fb02bdb5f54b1699749e3c8d588a5e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.800805 11643 tablet_service.cc:1467] Processing CreateTablet for tablet 93602521de174997bda6702d25aafce1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=98fb02bdb5f54b1699749e3c8d588a5e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.801043 11512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93602521de174997bda6702d25aafce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.801116 11643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93602521de174997bda6702d25aafce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.801201 11380 tablet_service.cc:1467] Processing CreateTablet for tablet 93602521de174997bda6702d25aafce1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=98fb02bdb5f54b1699749e3c8d588a5e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.801478 11380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93602521de174997bda6702d25aafce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.803072 12340 tablet_bootstrap.cc:492] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:47.803655 11381 tablet_service.cc:1467] Processing CreateTablet for tablet 64613ea537c842f29d810e829278394e (DEFAULT_TABLE table=rebalance_test_table_1 [id=98fb02bdb5f54b1699749e3c8d588a5e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.803699 11512 tablet_service.cc:1467] Processing CreateTablet for tablet 64613ea537c842f29d810e829278394e (DEFAULT_TABLE table=rebalance_test_table_1 [id=98fb02bdb5f54b1699749e3c8d588a5e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.803918 11512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64613ea537c842f29d810e829278394e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.803920 11381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64613ea537c842f29d810e829278394e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.804580 11511 tablet_service.cc:1467] Processing CreateTablet for tablet b9431cf5853c4148ac38855b1c083d4f (DEFAULT_TABLE table=rebalance_test_table_1 [id=98fb02bdb5f54b1699749e3c8d588a5e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.804805 11511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9431cf5853c4148ac38855b1c083d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.805006 11380 tablet_service.cc:1467] Processing CreateTablet for tablet b9431cf5853c4148ac38855b1c083d4f (DEFAULT_TABLE table=rebalance_test_table_1 [id=98fb02bdb5f54b1699749e3c8d588a5e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.805195 12340 tablet_bootstrap.cc:654] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.805233 11380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9431cf5853c4148ac38855b1c083d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.806223 11250 tablet_service.cc:1467] Processing CreateTablet for tablet b9431cf5853c4148ac38855b1c083d4f (DEFAULT_TABLE table=rebalance_test_table_1 [id=98fb02bdb5f54b1699749e3c8d588a5e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.806516 11250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9431cf5853c4148ac38855b1c083d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.806769 11643 tablet_service.cc:1467] Processing CreateTablet for tablet 64613ea537c842f29d810e829278394e (DEFAULT_TABLE table=rebalance_test_table_1 [id=98fb02bdb5f54b1699749e3c8d588a5e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:47.807154 11643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64613ea537c842f29d810e829278394e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:47.807595 12341 tablet_bootstrap.cc:492] T 93602521de174997bda6702d25aafce1 P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:47.808717 12341 tablet_bootstrap.cc:654] T 93602521de174997bda6702d25aafce1 P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.808868 12342 tablet_bootstrap.cc:492] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:47.809247 12344 tablet_bootstrap.cc:492] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:47.809733 12342 tablet_bootstrap.cc:654] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.809859 12341 tablet_bootstrap.cc:492] T 93602521de174997bda6702d25aafce1 P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:47.809957 12341 ts_tablet_manager.cc:1367] T 93602521de174997bda6702d25aafce1 P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:47.810145 12344 tablet_bootstrap.cc:654] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.810508 12341 raft_consensus.cc:348] T 93602521de174997bda6702d25aafce1 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.810626 12341 raft_consensus.cc:374] T 93602521de174997bda6702d25aafce1 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.810663 12341 raft_consensus.cc:729] T 93602521de174997bda6702d25aafce1 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.810770 12342 tablet_bootstrap.cc:492] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:47.810791 12341 consensus_queue.cc:260] T 93602521de174997bda6702d25aafce1 P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.810863 12342 ts_tablet_manager.cc:1367] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:47.811081 12341 ts_tablet_manager.cc:1398] T 93602521de174997bda6702d25aafce1 P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:47.811268 12341 tablet_bootstrap.cc:492] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:47.811275 12344 tablet_bootstrap.cc:492] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:47.811345 12344 ts_tablet_manager.cc:1367] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:47.811321 12342 raft_consensus.cc:348] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.811420 12342 raft_consensus.cc:374] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.811448 12342 raft_consensus.cc:729] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.811563 12342 consensus_queue.cc:260] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.811769 12344 raft_consensus.cc:348] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:47.811821 12342 ts_tablet_manager.cc:1398] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:47.811874 12344 raft_consensus.cc:374] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.811919 12344 raft_consensus.cc:729] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.812018 12342 tablet_bootstrap.cc:492] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:47.812714 12341 tablet_bootstrap.cc:654] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.812700 12340 tablet_bootstrap.cc:492] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:47.812804 12340 ts_tablet_manager.cc:1367] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 07:59:47.813218 12342 tablet_bootstrap.cc:654] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.813251 12340 raft_consensus.cc:348] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.813352 12340 raft_consensus.cc:374] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.813382 12340 raft_consensus.cc:729] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.813428 12344 consensus_queue.cc:260] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:47.813555 12340 consensus_queue.cc:260] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.813710 12344 ts_tablet_manager.cc:1398] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:47.813688 12262 raft_consensus.cc:482] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:47.813778 12262 raft_consensus.cc:504] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.813901 12340 ts_tablet_manager.cc:1398] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:47.814070 12340 tablet_bootstrap.cc:492] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:47.814863 12341 tablet_bootstrap.cc:492] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:47.814947 12341 ts_tablet_manager.cc:1367] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:47.814950 12340 tablet_bootstrap.cc:654] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.815006 12262 leader_election.cc:288] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:47.815438 12341 raft_consensus.cc:348] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.815552 12341 raft_consensus.cc:374] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.815583 12341 raft_consensus.cc:729] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.815703 12341 consensus_queue.cc:260] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.815972 12341 ts_tablet_manager.cc:1398] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:47.816421 12340 tablet_bootstrap.cc:492] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:47.816656 12340 ts_tablet_manager.cc:1367] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:59:47.817260 12340 raft_consensus.cc:348] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:47.817369 12340 raft_consensus.cc:374] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.817397 12340 raft_consensus.cc:729] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.817509 12340 consensus_queue.cc:260] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:47.817747 12340 ts_tablet_manager.cc:1398] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:47.817880 12340 tablet_bootstrap.cc:492] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:47.819242 12340 tablet_bootstrap.cc:654] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.820686 12342 tablet_bootstrap.cc:492] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:47.820816 12342 ts_tablet_manager.cc:1367] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:47.821295 12342 raft_consensus.cc:348] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.821420 12342 raft_consensus.cc:374] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.821451 12342 raft_consensus.cc:729] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.821578 12342 consensus_queue.cc:260] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.821847 12342 ts_tablet_manager.cc:1398] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:47.822027 12342 tablet_bootstrap.cc:492] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:47.823043 12342 tablet_bootstrap.cc:654] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:47.823014 11400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93602521de174997bda6702d25aafce1" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835" is_pre_election: true
I20240501 07:59:47.823164 11400 raft_consensus.cc:2454] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 0.
I20240501 07:59:47.823436 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93602521de174997bda6702d25aafce1" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2" is_pre_election: true
I20240501 07:59:47.823582 11662 raft_consensus.cc:2454] T 93602521de174997bda6702d25aafce1 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 0.
I20240501 07:59:47.823577 11465 leader_election.cc:302] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 655ebda1fb2c49c4af7087a760529835; no voters:
I20240501 07:59:47.823951 12340 tablet_bootstrap.cc:492] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:47.824060 12340 ts_tablet_manager.cc:1367] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 07:59:47.824143 12262 raft_consensus.cc:2781] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:47.824203 12262 raft_consensus.cc:482] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:47.824247 12262 raft_consensus.cc:3037] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.824550 12340 raft_consensus.cc:348] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.824662 12340 raft_consensus.cc:374] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.824695 12340 raft_consensus.cc:729] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.824889 12340 consensus_queue.cc:260] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.825176 12340 ts_tablet_manager.cc:1398] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 07:59:47.825311 12262 raft_consensus.cc:504] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.825667 12262 leader_election.cc:288] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 election: Requested vote from peers 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:47.825770 12342 tablet_bootstrap.cc:492] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:47.825856 12342 ts_tablet_manager.cc:1367] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:47.826085 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93602521de174997bda6702d25aafce1" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 07:59:47.826138 11400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93602521de174997bda6702d25aafce1" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 07:59:47.826218 11662 raft_consensus.cc:3037] T 93602521de174997bda6702d25aafce1 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.826232 11400 raft_consensus.cc:3037] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.826890 12342 raft_consensus.cc:348] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:47.826978 12342 raft_consensus.cc:374] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:47.827008 12342 raft_consensus.cc:729] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:47.827134 12342 consensus_queue.cc:260] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:47.827198 11662 raft_consensus.cc:2454] T 93602521de174997bda6702d25aafce1 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 1.
I20240501 07:59:47.827198 11400 raft_consensus.cc:2454] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 1.
I20240501 07:59:47.827548 12342 ts_tablet_manager.cc:1398] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:47.827828 11465 leader_election.cc:302] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 603919bb02364c04b6c530245b8b02f2; no voters:
I20240501 07:59:47.828008 12262 raft_consensus.cc:2781] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:47.828107 12262 raft_consensus.cc:686] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Becoming Leader. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Running, Role: LEADER
I20240501 07:59:47.828440 12262 consensus_queue.cc:237] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.831161 11127 catalog_manager.cc:5484] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 07:59:47.862833 12263 raft_consensus.cc:482] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:47.862998 12263 raft_consensus.cc:504] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.863574 12263 leader_election.cc:288] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:47.864208 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64613ea537c842f29d810e829278394e" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2" is_pre_election: true
I20240501 07:59:47.864372 11662 raft_consensus.cc:2454] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 0.
I20240501 07:59:47.864670 11334 leader_election.cc:302] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, 655ebda1fb2c49c4af7087a760529835; no voters:
I20240501 07:59:47.864820 12263 raft_consensus.cc:2781] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:47.864872 12263 raft_consensus.cc:482] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:47.864908 12263 raft_consensus.cc:3037] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.865712 12263 raft_consensus.cc:504] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.866065 12263 leader_election.cc:288] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 election: Requested vote from peers 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:47.866459 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64613ea537c842f29d810e829278394e" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 07:59:47.866583 11662 raft_consensus.cc:3037] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.867414 11662 raft_consensus.cc:2454] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 1.
I20240501 07:59:47.867569 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64613ea537c842f29d810e829278394e" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6" is_pre_election: true
I20240501 07:59:47.867579 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64613ea537c842f29d810e829278394e" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
I20240501 07:59:47.867730 11532 raft_consensus.cc:2454] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 0.
I20240501 07:59:47.867945 11334 leader_election.cc:302] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, 655ebda1fb2c49c4af7087a760529835; no voters:
I20240501 07:59:47.867990 12263 raft_consensus.cc:482] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:47.868106 12276 raft_consensus.cc:2781] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:47.868086 12263 raft_consensus.cc:504] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:47.868242 12276 raft_consensus.cc:686] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Becoming Leader. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Running, Role: LEADER
I20240501 07:59:47.868408 12263 leader_election.cc:288] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:47.868386 12276 consensus_queue.cc:237] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:47.868870 11270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9431cf5853c4148ac38855b1c083d4f" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" is_pre_election: true
I20240501 07:59:47.869011 11270 raft_consensus.cc:2454] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 0.
I20240501 07:59:47.869290 11335 leader_election.cc:302] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:47.869464 12276 raft_consensus.cc:2781] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:47.869427 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9431cf5853c4148ac38855b1c083d4f" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6" is_pre_election: true
I20240501 07:59:47.869513 12276 raft_consensus.cc:482] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:47.869546 12276 raft_consensus.cc:3037] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.869553 11531 raft_consensus.cc:2454] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 0.
I20240501 07:59:47.870072 11127 catalog_manager.cc:5484] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 reported cstate change: term changed from 0 to 1, leader changed from <none> to 655ebda1fb2c49c4af7087a760529835 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 07:59:47.870491 12276 raft_consensus.cc:504] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:47.870841 12276 leader_election.cc:288] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 election: Requested vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:47.871196 11270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9431cf5853c4148ac38855b1c083d4f" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 07:59:47.871326 11270 raft_consensus.cc:3037] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.871335 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9431cf5853c4148ac38855b1c083d4f" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
I20240501 07:59:47.871459 11531 raft_consensus.cc:3037] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:47.872251 11270 raft_consensus.cc:2454] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 1.
I20240501 07:59:47.872300 11531 raft_consensus.cc:2454] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 1.
I20240501 07:59:47.872589 11335 leader_election.cc:302] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:47.872781 12276 raft_consensus.cc:2781] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:47.872877 12276 raft_consensus.cc:686] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Becoming Leader. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Running, Role: LEADER
I20240501 07:59:47.873018 12276 consensus_queue.cc:237] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:47.874614 11124 catalog_manager.cc:5484] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 reported cstate change: term changed from 0 to 1, leader changed from <none> to 655ebda1fb2c49c4af7087a760529835 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
W20240501 07:59:48.001444 12362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:48.001744 12362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:48.005061 12362 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 07:59:48.090378 11400 raft_consensus.cc:1261] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:48.091203 11662 raft_consensus.cc:1261] T 93602521de174997bda6702d25aafce1 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:48.091549 12350 consensus_queue.cc:1035] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:48.092317 12350 consensus_queue.cc:1035] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:48.094004 11662 raft_consensus.cc:1261] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:48.094493 11531 raft_consensus.cc:3037] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.095620 11531 raft_consensus.cc:1261] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:48.096127 12359 consensus_queue.cc:1035] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:48.102677 12276 consensus_queue.cc:1035] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:48.115033 11270 raft_consensus.cc:1261] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:48.115664 12359 consensus_queue.cc:1035] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:48.116461 11532 raft_consensus.cc:1261] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:48.117062 12276 consensus_queue.cc:1035] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 106.856 ms
time per row: 0.0534282 ms
I20240501 07:59:48.161887 11124 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49370:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:48.168135 11250 tablet_service.cc:1467] Processing CreateTablet for tablet f2e63d24d7df4fa3b2b26695e70ebe9f (DEFAULT_TABLE table=rebalance_test_table_2 [id=69f109081c7449be8819974c5363fa32]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.168308 11512 tablet_service.cc:1467] Processing CreateTablet for tablet bd8348990f1b496bb62d8c0484193442 (DEFAULT_TABLE table=rebalance_test_table_2 [id=69f109081c7449be8819974c5363fa32]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.168464 11250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2e63d24d7df4fa3b2b26695e70ebe9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.168469 11643 tablet_service.cc:1467] Processing CreateTablet for tablet f2e63d24d7df4fa3b2b26695e70ebe9f (DEFAULT_TABLE table=rebalance_test_table_2 [id=69f109081c7449be8819974c5363fa32]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.168646 11512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8348990f1b496bb62d8c0484193442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.168728 11643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2e63d24d7df4fa3b2b26695e70ebe9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.170208 12344 tablet_bootstrap.cc:492] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:48.170770 12341 tablet_bootstrap.cc:492] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:48.171280 12344 tablet_bootstrap.cc:654] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.171581 12341 tablet_bootstrap.cc:654] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.171712 11643 tablet_service.cc:1467] Processing CreateTablet for tablet bd8348990f1b496bb62d8c0484193442 (DEFAULT_TABLE table=rebalance_test_table_2 [id=69f109081c7449be8819974c5363fa32]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.171712 11642 tablet_service.cc:1467] Processing CreateTablet for tablet 25cee7e51c7c438aa721d71fb82dc71e (DEFAULT_TABLE table=rebalance_test_table_2 [id=69f109081c7449be8819974c5363fa32]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.171967 11643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8348990f1b496bb62d8c0484193442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.172087 11642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25cee7e51c7c438aa721d71fb82dc71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.172207 12340 tablet_bootstrap.cc:492] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:48.172361 11381 tablet_service.cc:1467] Processing CreateTablet for tablet 25cee7e51c7c438aa721d71fb82dc71e (DEFAULT_TABLE table=rebalance_test_table_2 [id=69f109081c7449be8819974c5363fa32]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.172629 11381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25cee7e51c7c438aa721d71fb82dc71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.173070 12340 tablet_bootstrap.cc:654] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.173856 11380 tablet_service.cc:1467] Processing CreateTablet for tablet f2e63d24d7df4fa3b2b26695e70ebe9f (DEFAULT_TABLE table=rebalance_test_table_2 [id=69f109081c7449be8819974c5363fa32]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.174088 11380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2e63d24d7df4fa3b2b26695e70ebe9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.174721 11250 tablet_service.cc:1467] Processing CreateTablet for tablet bd8348990f1b496bb62d8c0484193442 (DEFAULT_TABLE table=rebalance_test_table_2 [id=69f109081c7449be8819974c5363fa32]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.174736 11248 tablet_service.cc:1467] Processing CreateTablet for tablet 25cee7e51c7c438aa721d71fb82dc71e (DEFAULT_TABLE table=rebalance_test_table_2 [id=69f109081c7449be8819974c5363fa32]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.174983 11250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8348990f1b496bb62d8c0484193442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.175233 11248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25cee7e51c7c438aa721d71fb82dc71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.175757 12344 tablet_bootstrap.cc:492] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:48.175843 12344 ts_tablet_manager.cc:1367] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:48.176337 12344 raft_consensus.cc:348] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:48.176436 12344 raft_consensus.cc:374] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.176463 12344 raft_consensus.cc:729] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.176595 12344 consensus_queue.cc:260] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:48.176649 12340 tablet_bootstrap.cc:492] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:48.176733 12340 ts_tablet_manager.cc:1367] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 07:59:48.176878 12344 ts_tablet_manager.cc:1398] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:48.176887 12342 tablet_bootstrap.cc:492] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:48.177029 12344 tablet_bootstrap.cc:492] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:48.177191 12340 raft_consensus.cc:348] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:48.177291 12340 raft_consensus.cc:374] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.177321 12340 raft_consensus.cc:729] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.177445 12340 consensus_queue.cc:260] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:48.177758 12342 tablet_bootstrap.cc:654] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.177718 12340 ts_tablet_manager.cc:1398] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 07:59:48.178400 12344 tablet_bootstrap.cc:654] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.179265 12341 tablet_bootstrap.cc:492] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:48.179343 12341 ts_tablet_manager.cc:1367] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 07:59:48.179739 12341 raft_consensus.cc:348] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:48.179841 12341 raft_consensus.cc:374] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.179868 12341 raft_consensus.cc:729] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.179986 12341 consensus_queue.cc:260] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:48.180266 12341 ts_tablet_manager.cc:1398] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:48.180442 12341 tablet_bootstrap.cc:492] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:48.180646 12342 tablet_bootstrap.cc:492] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:48.180734 12342 ts_tablet_manager.cc:1367] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:48.181174 12342 raft_consensus.cc:348] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:48.181238 12341 tablet_bootstrap.cc:654] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.181257 12342 raft_consensus.cc:374] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.181277 12342 raft_consensus.cc:729] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.181385 12342 consensus_queue.cc:260] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:48.181620 12342 ts_tablet_manager.cc:1398] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:48.181761 12342 tablet_bootstrap.cc:492] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:48.182521 12342 tablet_bootstrap.cc:654] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.182782 12344 tablet_bootstrap.cc:492] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:48.182866 12344 ts_tablet_manager.cc:1367] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 07:59:48.183012 12341 tablet_bootstrap.cc:492] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:48.183101 12341 ts_tablet_manager.cc:1367] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:48.183329 12344 raft_consensus.cc:348] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:48.183431 12344 raft_consensus.cc:374] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.183462 12344 raft_consensus.cc:729] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.183537 12341 raft_consensus.cc:348] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:48.183645 12341 raft_consensus.cc:374] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.183616 12344 consensus_queue.cc:260] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:48.183676 12341 raft_consensus.cc:729] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.183782 12341 consensus_queue.cc:260] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:48.183969 12344 ts_tablet_manager.cc:1398] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:48.184065 12341 ts_tablet_manager.cc:1398] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:48.184156 12344 tablet_bootstrap.cc:492] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:48.184260 12341 tablet_bootstrap.cc:492] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:48.185073 12344 tablet_bootstrap.cc:654] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.185155 12341 tablet_bootstrap.cc:654] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.186060 12342 tablet_bootstrap.cc:492] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:48.186161 12342 ts_tablet_manager.cc:1367] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 07:59:48.186481 12341 tablet_bootstrap.cc:492] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:48.186558 12341 ts_tablet_manager.cc:1367] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:48.186755 12342 raft_consensus.cc:348] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:48.186858 12342 raft_consensus.cc:374] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.186887 12342 raft_consensus.cc:729] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.186947 12341 raft_consensus.cc:348] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:48.187028 12341 raft_consensus.cc:374] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.187016 12342 consensus_queue.cc:260] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:48.187057 12341 raft_consensus.cc:729] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.187177 12341 consensus_queue.cc:260] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:48.187306 12342 ts_tablet_manager.cc:1398] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:48.187434 12341 ts_tablet_manager.cc:1398] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:48.188115 12344 tablet_bootstrap.cc:492] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:48.188261 12344 ts_tablet_manager.cc:1367] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 07:59:48.188832 12344 raft_consensus.cc:348] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:48.189009 12344 raft_consensus.cc:374] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.189088 12344 raft_consensus.cc:729] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.189236 12344 consensus_queue.cc:260] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:48.189510 12344 ts_tablet_manager.cc:1398] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:48.383744 12312 raft_consensus.cc:482] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:48.383893 12312 raft_consensus.cc:504] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:48.384349 12312 leader_election.cc:288] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:48.384814 11400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25cee7e51c7c438aa721d71fb82dc71e" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835" is_pre_election: true
I20240501 07:59:48.384910 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25cee7e51c7c438aa721d71fb82dc71e" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2" is_pre_election: true
I20240501 07:59:48.384976 11400 raft_consensus.cc:2454] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 0.
I20240501 07:59:48.385038 11662 raft_consensus.cc:2454] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 0.
I20240501 07:59:48.385299 11203 leader_election.cc:302] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:48.385457 12312 raft_consensus.cc:2781] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:48.385512 12312 raft_consensus.cc:482] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:48.385556 12312 raft_consensus.cc:3037] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.386353 12312 raft_consensus.cc:504] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:48.386711 12312 leader_election.cc:288] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 election: Requested vote from peers 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:48.387172 11400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25cee7e51c7c438aa721d71fb82dc71e" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 07:59:48.387176 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25cee7e51c7c438aa721d71fb82dc71e" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 07:59:48.387315 11662 raft_consensus.cc:3037] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.387315 11400 raft_consensus.cc:3037] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.388295 11662 raft_consensus.cc:2454] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 1.
I20240501 07:59:48.388295 11400 raft_consensus.cc:2454] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 1.
I20240501 07:59:48.388636 11203 leader_election.cc:302] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:48.388813 12312 raft_consensus.cc:2781] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:48.388975 12312 raft_consensus.cc:686] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Becoming Leader. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Running, Role: LEADER
I20240501 07:59:48.389127 12312 consensus_queue.cc:237] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:48.390841 11119 catalog_manager.cc:5484] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 07:59:48.394832 12312 raft_consensus.cc:482] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:48.394964 12312 raft_consensus.cc:504] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:48.395351 12312 leader_election.cc:288] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:48.395830 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8348990f1b496bb62d8c0484193442" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2" is_pre_election: true
I20240501 07:59:48.395853 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8348990f1b496bb62d8c0484193442" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6" is_pre_election: true
I20240501 07:59:48.395968 11531 raft_consensus.cc:2454] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 0.
I20240501 07:59:48.395973 11662 raft_consensus.cc:2454] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 0.
I20240501 07:59:48.396294 11203 leader_election.cc:302] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 603919bb02364c04b6c530245b8b02f2, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:48.396468 12312 raft_consensus.cc:2781] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:48.396525 12312 raft_consensus.cc:482] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:48.396551 12312 raft_consensus.cc:3037] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.397361 12312 raft_consensus.cc:504] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:48.397734 12312 leader_election.cc:288] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 election: Requested vote from peers 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:48.398121 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8348990f1b496bb62d8c0484193442" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
I20240501 07:59:48.398257 11531 raft_consensus.cc:3037] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.398238 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8348990f1b496bb62d8c0484193442" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 07:59:48.398362 11662 raft_consensus.cc:3037] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.399189 11662 raft_consensus.cc:2454] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 1.
I20240501 07:59:48.399202 11531 raft_consensus.cc:2454] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 1.
I20240501 07:59:48.399519 11205 leader_election.cc:302] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:48.399725 12312 raft_consensus.cc:2781] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:48.399820 12312 raft_consensus.cc:686] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Becoming Leader. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Running, Role: LEADER
I20240501 07:59:48.399989 12312 consensus_queue.cc:237] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:48.401751 11119 catalog_manager.cc:5484] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 07:59:48.471118 12312 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:48.471276 12312 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:48.472059 12312 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:48.472410 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2e63d24d7df4fa3b2b26695e70ebe9f" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2" is_pre_election: true
I20240501 07:59:48.472564 11662 raft_consensus.cc:2454] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 0.
I20240501 07:59:48.472535 11400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2e63d24d7df4fa3b2b26695e70ebe9f" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835" is_pre_election: true
I20240501 07:59:48.472666 11400 raft_consensus.cc:2454] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 0.
I20240501 07:59:48.472844 11203 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:48.473022 12312 raft_consensus.cc:2781] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:48.473074 12312 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:48.473125 12312 raft_consensus.cc:3037] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.473981 12312 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:48.474347 12312 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 election: Requested vote from peers 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:48.474766 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2e63d24d7df4fa3b2b26695e70ebe9f" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 07:59:48.474910 11662 raft_consensus.cc:3037] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.475060 11400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2e63d24d7df4fa3b2b26695e70ebe9f" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 07:59:48.475185 11400 raft_consensus.cc:3037] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.475880 11662 raft_consensus.cc:2454] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 1.
I20240501 07:59:48.476027 11400 raft_consensus.cc:2454] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 1.
I20240501 07:59:48.476392 11203 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:48.476675 12312 raft_consensus.cc:2781] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:48.476754 12312 raft_consensus.cc:686] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Becoming Leader. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Running, Role: LEADER
I20240501 07:59:48.476894 12312 consensus_queue.cc:237] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:48.478801 11119 catalog_manager.cc:5484] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
W20240501 07:59:48.659005 12411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:48.659307 12411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:48.662739 12411 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 07:59:48.752645 11400 raft_consensus.cc:1261] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:48.752866 11662 raft_consensus.cc:1261] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:48.753309 12407 consensus_queue.cc:1035] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:48.753386 11662 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:48.754362 11531 raft_consensus.cc:1261] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:48.754643 12312 consensus_queue.cc:1035] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:48.757117 11662 raft_consensus.cc:1261] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:48.757479 12443 consensus_queue.cc:1035] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:48.757717 12312 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:48.757807 12443 consensus_queue.cc:1035] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:48.757872 11401 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:48.758555 12407 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 99.2103 ms
time per row: 0.0496052 ms
I20240501 07:59:48.812031 11119 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49370:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:48.818406 11380 tablet_service.cc:1467] Processing CreateTablet for tablet 17bb1ff829bc47a6a0bd6030517652aa (DEFAULT_TABLE table=rebalance_test_table_3 [id=6447707eb34f45d7a8d3704cd09d7786]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.818552 11512 tablet_service.cc:1467] Processing CreateTablet for tablet 17bb1ff829bc47a6a0bd6030517652aa (DEFAULT_TABLE table=rebalance_test_table_3 [id=6447707eb34f45d7a8d3704cd09d7786]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.818805 11380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17bb1ff829bc47a6a0bd6030517652aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.818843 11512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17bb1ff829bc47a6a0bd6030517652aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.818925 11511 tablet_service.cc:1467] Processing CreateTablet for tablet 867ba08a15b44d4c9e6fa92b0d518d1d (DEFAULT_TABLE table=rebalance_test_table_3 [id=6447707eb34f45d7a8d3704cd09d7786]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.819149 11511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 867ba08a15b44d4c9e6fa92b0d518d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.821337 11511 tablet_service.cc:1467] Processing CreateTablet for tablet 48b0ff0769dc426190f038811155c741 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6447707eb34f45d7a8d3704cd09d7786]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.821599 11511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b0ff0769dc426190f038811155c741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.821869 11250 tablet_service.cc:1467] Processing CreateTablet for tablet 17bb1ff829bc47a6a0bd6030517652aa (DEFAULT_TABLE table=rebalance_test_table_3 [id=6447707eb34f45d7a8d3704cd09d7786]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.822124 11250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17bb1ff829bc47a6a0bd6030517652aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.822672 11248 tablet_service.cc:1467] Processing CreateTablet for tablet 867ba08a15b44d4c9e6fa92b0d518d1d (DEFAULT_TABLE table=rebalance_test_table_3 [id=6447707eb34f45d7a8d3704cd09d7786]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.822881 11248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 867ba08a15b44d4c9e6fa92b0d518d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.824401 11643 tablet_service.cc:1467] Processing CreateTablet for tablet 867ba08a15b44d4c9e6fa92b0d518d1d (DEFAULT_TABLE table=rebalance_test_table_3 [id=6447707eb34f45d7a8d3704cd09d7786]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.824659 11643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 867ba08a15b44d4c9e6fa92b0d518d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.824401 11642 tablet_service.cc:1467] Processing CreateTablet for tablet 48b0ff0769dc426190f038811155c741 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6447707eb34f45d7a8d3704cd09d7786]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.824867 11642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b0ff0769dc426190f038811155c741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.825418 11380 tablet_service.cc:1467] Processing CreateTablet for tablet 48b0ff0769dc426190f038811155c741 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6447707eb34f45d7a8d3704cd09d7786]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:48.825690 11380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b0ff0769dc426190f038811155c741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:48.827031 12475 tablet_bootstrap.cc:492] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:48.827641 12473 tablet_bootstrap.cc:492] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:48.827838 12474 tablet_bootstrap.cc:492] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:48.827955 12475 tablet_bootstrap.cc:654] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.828603 12473 tablet_bootstrap.cc:654] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.828620 12472 tablet_bootstrap.cc:492] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:48.828680 12474 tablet_bootstrap.cc:654] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.829407 12472 tablet_bootstrap.cc:654] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.829738 12474 tablet_bootstrap.cc:492] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:48.829738 12473 tablet_bootstrap.cc:492] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:48.829823 12474 ts_tablet_manager.cc:1367] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:48.829823 12473 ts_tablet_manager.cc:1367] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:48.830236 12474 raft_consensus.cc:348] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:48.830236 12473 raft_consensus.cc:348] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:48.830353 12473 raft_consensus.cc:374] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.830353 12474 raft_consensus.cc:374] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.830384 12473 raft_consensus.cc:729] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.830384 12474 raft_consensus.cc:729] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.830557 12475 tablet_bootstrap.cc:492] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:48.830543 12474 consensus_queue.cc:260] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:48.830557 12473 consensus_queue.cc:260] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:48.830649 12475 ts_tablet_manager.cc:1367] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:48.830821 12474 ts_tablet_manager.cc:1398] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:48.830964 12474 tablet_bootstrap.cc:492] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:48.831094 12475 raft_consensus.cc:348] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:48.831192 12475 raft_consensus.cc:374] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.831220 12475 raft_consensus.cc:729] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.831336 12475 consensus_queue.cc:260] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:48.831645 12475 ts_tablet_manager.cc:1398] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:48.831810 12474 tablet_bootstrap.cc:654] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.831840 12475 tablet_bootstrap.cc:492] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:48.832125 12473 ts_tablet_manager.cc:1398] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:48.832302 12473 tablet_bootstrap.cc:492] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:48.832914 12474 tablet_bootstrap.cc:492] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:48.832928 12475 tablet_bootstrap.cc:654] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.832999 12474 ts_tablet_manager.cc:1367] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:48.833140 12473 tablet_bootstrap.cc:654] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.833439 12474 raft_consensus.cc:348] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:48.833536 12474 raft_consensus.cc:374] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.833566 12474 raft_consensus.cc:729] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.833693 12474 consensus_queue.cc:260] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:48.833976 12474 ts_tablet_manager.cc:1398] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:48.834069 12472 tablet_bootstrap.cc:492] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:48.834167 12472 ts_tablet_manager.cc:1367] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 07:59:48.834316 12473 tablet_bootstrap.cc:492] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:48.834398 12473 ts_tablet_manager.cc:1367] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:48.834622 12472 raft_consensus.cc:348] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:48.834751 12472 raft_consensus.cc:374] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.834784 12472 raft_consensus.cc:729] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.834872 12473 raft_consensus.cc:348] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:48.834899 12472 consensus_queue.cc:260] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:48.834970 12473 raft_consensus.cc:374] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.835014 12473 raft_consensus.cc:729] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.835127 12473 consensus_queue.cc:260] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:48.835158 12472 ts_tablet_manager.cc:1398] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:48.835330 12472 tablet_bootstrap.cc:492] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:48.835372 12473 ts_tablet_manager.cc:1398] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:48.836196 12472 tablet_bootstrap.cc:654] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.836535 12475 tablet_bootstrap.cc:492] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:48.836638 12475 ts_tablet_manager.cc:1367] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:48.837101 12475 raft_consensus.cc:348] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:48.837222 12475 raft_consensus.cc:374] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.837256 12475 raft_consensus.cc:729] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.837374 12475 consensus_queue.cc:260] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:48.837675 12475 ts_tablet_manager.cc:1398] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:48.837981 12472 tablet_bootstrap.cc:492] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:48.838069 12472 ts_tablet_manager.cc:1367] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:48.838511 12472 raft_consensus.cc:348] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:48.838618 12472 raft_consensus.cc:374] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.838649 12472 raft_consensus.cc:729] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.838757 12472 consensus_queue.cc:260] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:48.839026 12472 ts_tablet_manager.cc:1398] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:48.839179 12472 tablet_bootstrap.cc:492] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:48.840011 12472 tablet_bootstrap.cc:654] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:48.841286 12472 tablet_bootstrap.cc:492] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:48.841367 12472 ts_tablet_manager.cc:1367] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:48.841753 12472 raft_consensus.cc:348] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:48.841847 12472 raft_consensus.cc:374] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:48.841876 12472 raft_consensus.cc:729] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:48.841995 12472 consensus_queue.cc:260] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:48.842247 12472 ts_tablet_manager.cc:1398] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:48.991736 12262 raft_consensus.cc:482] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:48.991896 12262 raft_consensus.cc:504] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:48.992329 12262 leader_election.cc:288] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:48.992834 12441 raft_consensus.cc:482] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:48.992863 11400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b0ff0769dc426190f038811155c741" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835" is_pre_election: true
I20240501 07:59:48.992926 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b0ff0769dc426190f038811155c741" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2" is_pre_election: true
I20240501 07:59:48.992949 12441 raft_consensus.cc:504] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:48.993006 11400 raft_consensus.cc:2454] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 0.
I20240501 07:59:48.993065 11662 raft_consensus.cc:2454] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 0.
I20240501 07:59:48.993358 12441 leader_election.cc:288] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:48.993352 11465 leader_election.cc:302] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 603919bb02364c04b6c530245b8b02f2; no voters:
I20240501 07:59:48.993520 12262 raft_consensus.cc:2781] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:48.993584 12262 raft_consensus.cc:482] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:48.993620 12262 raft_consensus.cc:3037] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.993816 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867ba08a15b44d4c9e6fa92b0d518d1d" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2" is_pre_election: true
I20240501 07:59:48.993821 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867ba08a15b44d4c9e6fa92b0d518d1d" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6" is_pre_election: true
I20240501 07:59:48.993916 11531 raft_consensus.cc:2454] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 0.
I20240501 07:59:48.993923 11662 raft_consensus.cc:2454] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 0.
I20240501 07:59:48.994208 11205 leader_election.cc:302] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:48.994390 12441 raft_consensus.cc:2781] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:48.994490 12441 raft_consensus.cc:482] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:48.994580 12441 raft_consensus.cc:3037] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.994710 12262 raft_consensus.cc:504] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:48.995007 12262 leader_election.cc:288] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 election: Requested vote from peers 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:48.995376 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b0ff0769dc426190f038811155c741" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 07:59:48.995393 11400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b0ff0769dc426190f038811155c741" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 07:59:48.995473 11662 raft_consensus.cc:3037] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.995507 11400 raft_consensus.cc:3037] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.995795 12441 raft_consensus.cc:504] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:48.996153 12441 leader_election.cc:288] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 election: Requested vote from peers 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:48.996260 11662 raft_consensus.cc:2454] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 1.
I20240501 07:59:48.996435 11400 raft_consensus.cc:2454] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 1.
I20240501 07:59:48.996539 11465 leader_election.cc:302] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 603919bb02364c04b6c530245b8b02f2; no voters:
I20240501 07:59:48.996855 12262 raft_consensus.cc:2781] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:48.996881 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867ba08a15b44d4c9e6fa92b0d518d1d" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 07:59:48.996887 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867ba08a15b44d4c9e6fa92b0d518d1d" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
I20240501 07:59:48.996987 11531 raft_consensus.cc:3037] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.997004 11662 raft_consensus.cc:3037] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:48.997045 12262 raft_consensus.cc:686] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Becoming Leader. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Running, Role: LEADER
I20240501 07:59:48.997192 12262 consensus_queue.cc:237] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:48.997856 11531 raft_consensus.cc:2454] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 1.
I20240501 07:59:48.998270 11205 leader_election.cc:302] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:48.998562 12441 raft_consensus.cc:2781] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:48.998874 11662 raft_consensus.cc:2454] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 1.
I20240501 07:59:48.999104 12441 raft_consensus.cc:686] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Becoming Leader. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Running, Role: LEADER
I20240501 07:59:48.999264 12441 consensus_queue.cc:237] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:49.000483 11119 catalog_manager.cc:5484] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 07:59:49.001009 11122 catalog_manager.cc:5484] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 07:59:49.144027 12485 raft_consensus.cc:482] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:49.144181 12485 raft_consensus.cc:504] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:49.144707 12485 leader_election.cc:288] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:49.145182 11400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17bb1ff829bc47a6a0bd6030517652aa" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835" is_pre_election: true
I20240501 07:59:49.145349 11400 raft_consensus.cc:2454] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 0.
I20240501 07:59:49.145666 11465 leader_election.cc:302] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 655ebda1fb2c49c4af7087a760529835; no voters:
I20240501 07:59:49.145857 12485 raft_consensus.cc:2781] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:49.145925 12485 raft_consensus.cc:482] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:49.145964 12485 raft_consensus.cc:3037] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:49.147464 12485 raft_consensus.cc:504] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:49.147855 12485 leader_election.cc:288] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 election: Requested vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:49.148237 11400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17bb1ff829bc47a6a0bd6030517652aa" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 07:59:49.148372 11400 raft_consensus.cc:3037] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:49.149077 11270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17bb1ff829bc47a6a0bd6030517652aa" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" is_pre_election: true
I20240501 07:59:49.149147 11269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17bb1ff829bc47a6a0bd6030517652aa" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 07:59:49.149226 11270 raft_consensus.cc:2454] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 0.
I20240501 07:59:49.149327 11400 raft_consensus.cc:2454] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 1.
I20240501 07:59:49.149613 11465 leader_election.cc:302] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 655ebda1fb2c49c4af7087a760529835; no voters:
I20240501 07:59:49.149852 12485 raft_consensus.cc:2781] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:49.149938 12485 raft_consensus.cc:686] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Becoming Leader. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Running, Role: LEADER
I20240501 07:59:49.150071 12485 consensus_queue.cc:237] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:49.151849 11122 catalog_manager.cc:5484] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
W20240501 07:59:49.305091 12491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:49.305409 12491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:49.309180 12491 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 07:59:49.409606 11269 raft_consensus.cc:3037] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:49.409794 11400 raft_consensus.cc:1261] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:49.410233 12485 consensus_queue.cc:1035] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:49.410874 11269 raft_consensus.cc:1261] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:49.412046 11531 raft_consensus.cc:1261] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:49.412418 11663 raft_consensus.cc:1261] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:49.412416 12262 consensus_queue.cc:1035] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:49.412935 12442 consensus_queue.cc:1035] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:49.413224 12442 consensus_queue.cc:1035] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:49.414274 11663 raft_consensus.cc:1261] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:49.414831 11401 raft_consensus.cc:1261] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:49.415596 12485 consensus_queue.cc:1035] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:49.416332 12262 consensus_queue.cc:1035] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 121.454 ms
time per row: 0.060727 ms
I20240501 07:59:49.477340 11121 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49370:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:49.483215 11250 tablet_service.cc:1467] Processing CreateTablet for tablet 7a266f266c3d4fd49fa8b325f1045d61 (DEFAULT_TABLE table=rebalance_test_table_4 [id=384b2b245c6b4931a4fc02c6b5d647b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:49.483565 11250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a266f266c3d4fd49fa8b325f1045d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:49.483593 11511 tablet_service.cc:1467] Processing CreateTablet for tablet b4707b952d174a9bb877ad8bd7a11268 (DEFAULT_TABLE table=rebalance_test_table_4 [id=384b2b245c6b4931a4fc02c6b5d647b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:49.483845 11511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4707b952d174a9bb877ad8bd7a11268. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:49.484149 11380 tablet_service.cc:1467] Processing CreateTablet for tablet 7a266f266c3d4fd49fa8b325f1045d61 (DEFAULT_TABLE table=rebalance_test_table_4 [id=384b2b245c6b4931a4fc02c6b5d647b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:49.484411 11380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a266f266c3d4fd49fa8b325f1045d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:49.486137 11381 tablet_service.cc:1467] Processing CreateTablet for tablet b4707b952d174a9bb877ad8bd7a11268 (DEFAULT_TABLE table=rebalance_test_table_4 [id=384b2b245c6b4931a4fc02c6b5d647b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:49.486361 11381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4707b952d174a9bb877ad8bd7a11268. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:49.486472 11642 tablet_service.cc:1467] Processing CreateTablet for tablet 7a266f266c3d4fd49fa8b325f1045d61 (DEFAULT_TABLE table=rebalance_test_table_4 [id=384b2b245c6b4931a4fc02c6b5d647b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:49.486754 11642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a266f266c3d4fd49fa8b325f1045d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:49.486989 11643 tablet_service.cc:1467] Processing CreateTablet for tablet f80db2e37bbc4716ad3a5200c61ee97b (DEFAULT_TABLE table=rebalance_test_table_4 [id=384b2b245c6b4931a4fc02c6b5d647b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:49.487193 11643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80db2e37bbc4716ad3a5200c61ee97b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:49.489114 12557 tablet_bootstrap.cc:492] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:49.489611 11511 tablet_service.cc:1467] Processing CreateTablet for tablet f80db2e37bbc4716ad3a5200c61ee97b (DEFAULT_TABLE table=rebalance_test_table_4 [id=384b2b245c6b4931a4fc02c6b5d647b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:49.489861 11511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80db2e37bbc4716ad3a5200c61ee97b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:49.490087 12557 tablet_bootstrap.cc:654] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:49.490159 11250 tablet_service.cc:1467] Processing CreateTablet for tablet b4707b952d174a9bb877ad8bd7a11268 (DEFAULT_TABLE table=rebalance_test_table_4 [id=384b2b245c6b4931a4fc02c6b5d647b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:49.490476 11250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4707b952d174a9bb877ad8bd7a11268. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:49.491248 12557 tablet_bootstrap.cc:492] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:49.491331 12557 ts_tablet_manager.cc:1367] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:49.491812 12557 raft_consensus.cc:348] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:49.491920 12557 raft_consensus.cc:374] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:49.491958 12557 raft_consensus.cc:729] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:49.492084 12557 consensus_queue.cc:260] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:49.492364 12557 ts_tablet_manager.cc:1398] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:49.492558 12557 tablet_bootstrap.cc:492] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:49.492749 11248 tablet_service.cc:1467] Processing CreateTablet for tablet f80db2e37bbc4716ad3a5200c61ee97b (DEFAULT_TABLE table=rebalance_test_table_4 [id=384b2b245c6b4931a4fc02c6b5d647b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:49.492910 12554 tablet_bootstrap.cc:492] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:49.492990 11248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80db2e37bbc4716ad3a5200c61ee97b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:49.493222 12556 tablet_bootstrap.cc:492] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:49.493422 12557 tablet_bootstrap.cc:654] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:49.494006 12554 tablet_bootstrap.cc:654] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:49.494019 12556 tablet_bootstrap.cc:654] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:49.495185 12555 tablet_bootstrap.cc:492] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:49.495319 12556 tablet_bootstrap.cc:492] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:49.495420 12556 ts_tablet_manager.cc:1367] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:49.495502 12557 tablet_bootstrap.cc:492] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:49.495591 12557 ts_tablet_manager.cc:1367] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:59:49.495880 12554 tablet_bootstrap.cc:492] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:49.495885 12556 raft_consensus.cc:348] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:49.496004 12556 raft_consensus.cc:374] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:49.496019 12555 tablet_bootstrap.cc:654] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:49.496039 12556 raft_consensus.cc:729] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:49.496019 12557 raft_consensus.cc:348] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:49.496131 12557 raft_consensus.cc:374] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:49.496176 12557 raft_consensus.cc:729] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:49.496178 12556 consensus_queue.cc:260] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:49.496165 12554 ts_tablet_manager.cc:1367] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:49.496301 12557 consensus_queue.cc:260] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:49.496452 12556 ts_tablet_manager.cc:1398] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:49.496562 12557 ts_tablet_manager.cc:1398] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:49.496634 12556 tablet_bootstrap.cc:492] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:49.496922 12554 raft_consensus.cc:348] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:49.497045 12554 raft_consensus.cc:374] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:49.497074 12554 raft_consensus.cc:729] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:49.497197 12554 consensus_queue.cc:260] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:49.497460 12554 ts_tablet_manager.cc:1398] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:49.497519 12556 tablet_bootstrap.cc:654] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:49.497650 12554 tablet_bootstrap.cc:492] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:49.498121 12555 tablet_bootstrap.cc:492] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:49.498217 12555 ts_tablet_manager.cc:1367] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:49.498493 12554 tablet_bootstrap.cc:654] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:49.498685 12555 raft_consensus.cc:348] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:49.498787 12555 raft_consensus.cc:374] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:49.498817 12555 raft_consensus.cc:729] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:49.498945 12555 consensus_queue.cc:260] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:49.499236 12555 ts_tablet_manager.cc:1398] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:49.499410 12555 tablet_bootstrap.cc:492] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:49.499451 12556 tablet_bootstrap.cc:492] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:49.499526 12556 ts_tablet_manager.cc:1367] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:49.499899 12556 raft_consensus.cc:348] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:49.499984 12556 raft_consensus.cc:374] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:49.500002 12556 raft_consensus.cc:729] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:49.500109 12556 consensus_queue.cc:260] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:49.500270 12555 tablet_bootstrap.cc:654] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:49.500375 12556 ts_tablet_manager.cc:1398] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:49.501080 12554 tablet_bootstrap.cc:492] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:49.501217 12554 ts_tablet_manager.cc:1367] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 07:59:49.501338 12555 tablet_bootstrap.cc:492] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:49.501428 12555 ts_tablet_manager.cc:1367] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:49.501749 12554 raft_consensus.cc:348] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:49.501864 12554 raft_consensus.cc:374] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:49.501818 12555 raft_consensus.cc:348] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:49.501904 12554 raft_consensus.cc:729] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:49.501909 12555 raft_consensus.cc:374] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:49.501941 12555 raft_consensus.cc:729] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:49.502053 12554 consensus_queue.cc:260] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:49.502053 12555 consensus_queue.cc:260] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:49.502367 12555 ts_tablet_manager.cc:1398] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:49.502490 12554 ts_tablet_manager.cc:1398] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:49.502686 12554 tablet_bootstrap.cc:492] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:49.503530 12554 tablet_bootstrap.cc:654] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:49.505528 12554 tablet_bootstrap.cc:492] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:49.505641 12554 ts_tablet_manager.cc:1367] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:49.506256 12554 raft_consensus.cc:348] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:49.506402 12554 raft_consensus.cc:374] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:49.506477 12554 raft_consensus.cc:729] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:49.506672 12554 consensus_queue.cc:260] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:49.507015 12554 ts_tablet_manager.cc:1398] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:49.515753 12441 raft_consensus.cc:482] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:49.515888 12441 raft_consensus.cc:504] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:49.516261 12441 leader_election.cc:288] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:49.516817 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a266f266c3d4fd49fa8b325f1045d61" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835" is_pre_election: true
I20240501 07:59:49.516817 11663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a266f266c3d4fd49fa8b325f1045d61" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2" is_pre_election: true
I20240501 07:59:49.516995 11663 raft_consensus.cc:2454] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 0.
I20240501 07:59:49.516988 11401 raft_consensus.cc:2454] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 0.
I20240501 07:59:49.517352 11203 leader_election.cc:302] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:49.517544 12441 raft_consensus.cc:2781] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:49.517602 12441 raft_consensus.cc:482] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:49.517647 12441 raft_consensus.cc:3037] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:49.518538 12441 raft_consensus.cc:504] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:49.518949 12441 leader_election.cc:288] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 election: Requested vote from peers 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:49.519352 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a266f266c3d4fd49fa8b325f1045d61" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 07:59:49.519352 11663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a266f266c3d4fd49fa8b325f1045d61" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 07:59:49.519481 11401 raft_consensus.cc:3037] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:49.519481 11663 raft_consensus.cc:3037] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:49.520444 11663 raft_consensus.cc:2454] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 1.
I20240501 07:59:49.520443 11401 raft_consensus.cc:2454] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 1.
I20240501 07:59:49.520771 11203 leader_election.cc:302] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:49.520934 12441 raft_consensus.cc:2781] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:49.521027 12441 raft_consensus.cc:686] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Becoming Leader. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Running, Role: LEADER
I20240501 07:59:49.521221 12441 consensus_queue.cc:237] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:49.523068 11128 catalog_manager.cc:5484] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 07:59:49.692821 12535 raft_consensus.cc:482] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:49.692979 12535 raft_consensus.cc:504] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:49.694000 12535 leader_election.cc:288] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957), e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 07:59:49.697611 11269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80db2e37bbc4716ad3a5200c61ee97b" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" is_pre_election: true
I20240501 07:59:49.697660 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80db2e37bbc4716ad3a5200c61ee97b" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6" is_pre_election: true
I20240501 07:59:49.697804 11531 raft_consensus.cc:2454] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 0.
I20240501 07:59:49.697804 11269 raft_consensus.cc:2454] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 0.
I20240501 07:59:49.698170 11598 leader_election.cc:302] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 603919bb02364c04b6c530245b8b02f2; no voters:
I20240501 07:59:49.698452 12535 raft_consensus.cc:2781] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:49.698518 12535 raft_consensus.cc:482] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:49.698563 12535 raft_consensus.cc:3037] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:49.699497 12535 raft_consensus.cc:504] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:49.699856 12535 leader_election.cc:288] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 election: Requested vote from peers 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957), e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 07:59:49.700158 11269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80db2e37bbc4716ad3a5200c61ee97b" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 07:59:49.700162 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80db2e37bbc4716ad3a5200c61ee97b" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
I20240501 07:59:49.700266 11269 raft_consensus.cc:3037] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:49.700299 11531 raft_consensus.cc:3037] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:49.701031 11269 raft_consensus.cc:2454] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 1.
I20240501 07:59:49.701036 11531 raft_consensus.cc:2454] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 1.
I20240501 07:59:49.701319 11597 leader_election.cc:302] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:49.701483 12535 raft_consensus.cc:2781] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:49.701575 12535 raft_consensus.cc:686] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Becoming Leader. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Running, Role: LEADER
I20240501 07:59:49.701905 12535 consensus_queue.cc:237] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:49.704301 11128 catalog_manager.cc:5484] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 603919bb02364c04b6c530245b8b02f2 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 07:59:49.837002 12524 raft_consensus.cc:482] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:49.837143 12524 raft_consensus.cc:504] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:49.837569 12524 leader_election.cc:288] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:49.837927 11269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4707b952d174a9bb877ad8bd7a11268" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" is_pre_election: true
I20240501 07:59:49.838038 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4707b952d174a9bb877ad8bd7a11268" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835" is_pre_election: true
I20240501 07:59:49.838089 11269 raft_consensus.cc:2454] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 0.
I20240501 07:59:49.838172 11401 raft_consensus.cc:2454] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 0.
I20240501 07:59:49.838487 11466 leader_election.cc:302] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:49.838662 12524 raft_consensus.cc:2781] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:49.838721 12524 raft_consensus.cc:482] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:49.838747 12524 raft_consensus.cc:3037] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:49.839639 12524 raft_consensus.cc:504] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:49.839967 12524 leader_election.cc:288] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 election: Requested vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:49.840376 11269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4707b952d174a9bb877ad8bd7a11268" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 07:59:49.840376 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4707b952d174a9bb877ad8bd7a11268" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 07:59:49.840518 11269 raft_consensus.cc:3037] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:49.840523 11401 raft_consensus.cc:3037] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:49.841462 11269 raft_consensus.cc:2454] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 1.
I20240501 07:59:49.841462 11401 raft_consensus.cc:2454] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 1.
I20240501 07:59:49.841809 11465 leader_election.cc:302] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 655ebda1fb2c49c4af7087a760529835; no voters:
I20240501 07:59:49.842005 12524 raft_consensus.cc:2781] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:49.842092 12524 raft_consensus.cc:686] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Becoming Leader. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Running, Role: LEADER
I20240501 07:59:49.842242 12524 consensus_queue.cc:237] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:49.843997 11128 catalog_manager.cc:5484] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 07:59:49.899844 12441 consensus_queue.cc:1035] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:49.903780 12441 consensus_queue.cc:1035] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 07:59:49.980471 12572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:49.980753 12572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:49.984098 12572 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 07:59:50.076238 11269 raft_consensus.cc:1261] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:50.076663 12524 consensus_queue.cc:1035] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:50.079906 11270 raft_consensus.cc:1261] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:50.080466 12530 consensus_queue.cc:1035] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:50.082535 11531 raft_consensus.cc:1261] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:50.083487 12535 consensus_queue.cc:1035] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:50.085423 11401 raft_consensus.cc:1261] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:50.086436 12524 consensus_queue.cc:1035] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:50.119791 12607 mvcc.cc:205] Tried to move back new op lower bound from 7022798397759983616 to 7022798396221665280. Current Snapshot: MvccSnapshot[applied={T|T < 7022798397759983616}]
INSERT report
rows total: 2000
time total: 118.962 ms
time per row: 0.0594812 ms
I20240501 07:59:50.147151 11123 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49370:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:50.153175 11381 tablet_service.cc:1467] Processing CreateTablet for tablet 90f11b5693d7456d8a604e6c8726e69a (DEFAULT_TABLE table=rebalance_test_table_5 [id=0af21d4c4fcb49a7ac0756a5ded5d57d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:50.153175 11643 tablet_service.cc:1467] Processing CreateTablet for tablet 90f11b5693d7456d8a604e6c8726e69a (DEFAULT_TABLE table=rebalance_test_table_5 [id=0af21d4c4fcb49a7ac0756a5ded5d57d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:50.153503 11381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f11b5693d7456d8a604e6c8726e69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.153503 11643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f11b5693d7456d8a604e6c8726e69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.153631 11248 tablet_service.cc:1467] Processing CreateTablet for tablet b21b6e2023e842739a7c0cc946e7c2fd (DEFAULT_TABLE table=rebalance_test_table_5 [id=0af21d4c4fcb49a7ac0756a5ded5d57d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:50.153868 11248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21b6e2023e842739a7c0cc946e7c2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.155843 12631 tablet_bootstrap.cc:492] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:50.156016 11380 tablet_service.cc:1467] Processing CreateTablet for tablet f61e70ba02384f81b9ae36b1c65465b9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=0af21d4c4fcb49a7ac0756a5ded5d57d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:50.156231 11380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61e70ba02384f81b9ae36b1c65465b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.156448 12633 tablet_bootstrap.cc:492] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:50.156980 12631 tablet_bootstrap.cc:654] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:50.157056 12632 tablet_bootstrap.cc:492] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:50.157325 12633 tablet_bootstrap.cc:654] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:50.157857 12632 tablet_bootstrap.cc:654] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:50.158340 11642 tablet_service.cc:1467] Processing CreateTablet for tablet f61e70ba02384f81b9ae36b1c65465b9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=0af21d4c4fcb49a7ac0756a5ded5d57d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:50.158521 12633 tablet_bootstrap.cc:492] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:50.158598 12633 ts_tablet_manager.cc:1367] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:50.158671 11642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61e70ba02384f81b9ae36b1c65465b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.158643 11641 tablet_service.cc:1467] Processing CreateTablet for tablet b21b6e2023e842739a7c0cc946e7c2fd (DEFAULT_TABLE table=rebalance_test_table_5 [id=0af21d4c4fcb49a7ac0756a5ded5d57d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:50.158824 11641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21b6e2023e842739a7c0cc946e7c2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.159049 12633 raft_consensus.cc:348] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:50.159157 12633 raft_consensus.cc:374] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:50.159183 12633 raft_consensus.cc:729] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:50.159303 12633 consensus_queue.cc:260] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:50.159562 12633 ts_tablet_manager.cc:1398] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:50.161751 11511 tablet_service.cc:1467] Processing CreateTablet for tablet 90f11b5693d7456d8a604e6c8726e69a (DEFAULT_TABLE table=rebalance_test_table_5 [id=0af21d4c4fcb49a7ac0756a5ded5d57d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:50.161785 11380 tablet_service.cc:1467] Processing CreateTablet for tablet b21b6e2023e842739a7c0cc946e7c2fd (DEFAULT_TABLE table=rebalance_test_table_5 [id=0af21d4c4fcb49a7ac0756a5ded5d57d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:50.161803 11512 tablet_service.cc:1467] Processing CreateTablet for tablet f61e70ba02384f81b9ae36b1c65465b9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=0af21d4c4fcb49a7ac0756a5ded5d57d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:50.162035 11380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21b6e2023e842739a7c0cc946e7c2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.162070 11511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f11b5693d7456d8a604e6c8726e69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.162226 11512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61e70ba02384f81b9ae36b1c65465b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.164678 12632 tablet_bootstrap.cc:492] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:50.164717 12637 tablet_bootstrap.cc:492] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:50.164790 12632 ts_tablet_manager.cc:1367] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 07:59:50.165231 12632 raft_consensus.cc:348] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:50.165364 12632 raft_consensus.cc:374] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:50.165395 12632 raft_consensus.cc:729] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:50.165530 12632 consensus_queue.cc:260] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:50.165751 12637 tablet_bootstrap.cc:654] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:50.165829 12632 ts_tablet_manager.cc:1398] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:50.165971 12632 tablet_bootstrap.cc:492] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:50.166898 12632 tablet_bootstrap.cc:654] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:50.167035 12637 tablet_bootstrap.cc:492] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:50.167124 12637 ts_tablet_manager.cc:1367] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:50.167517 12637 raft_consensus.cc:348] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:50.167618 12637 raft_consensus.cc:374] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:50.167646 12637 raft_consensus.cc:729] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:50.167771 12637 consensus_queue.cc:260] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:50.167827 12631 tablet_bootstrap.cc:492] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:50.167920 12631 ts_tablet_manager.cc:1367] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 07:59:50.168061 12637 ts_tablet_manager.cc:1398] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:50.168283 12637 tablet_bootstrap.cc:492] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:50.168403 12631 raft_consensus.cc:348] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:50.168512 12631 raft_consensus.cc:374] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:50.168543 12631 raft_consensus.cc:729] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:50.168673 12631 consensus_queue.cc:260] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:50.168958 12631 ts_tablet_manager.cc:1398] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:50.169144 12631 tablet_bootstrap.cc:492] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:50.169183 12637 tablet_bootstrap.cc:654] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:50.170023 12631 tablet_bootstrap.cc:654] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:50.170346 12637 tablet_bootstrap.cc:492] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:50.170457 12637 ts_tablet_manager.cc:1367] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:50.170840 12637 raft_consensus.cc:348] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:50.170948 12637 raft_consensus.cc:374] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:50.170981 12637 raft_consensus.cc:729] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:50.171113 12637 consensus_queue.cc:260] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:50.171377 12637 ts_tablet_manager.cc:1398] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:50.171826 12631 tablet_bootstrap.cc:492] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:50.171940 12631 ts_tablet_manager.cc:1367] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:50.172416 12631 raft_consensus.cc:348] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:50.172580 12631 raft_consensus.cc:374] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:50.172654 12631 raft_consensus.cc:729] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:50.172789 12631 consensus_queue.cc:260] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:50.173029 12631 ts_tablet_manager.cc:1398] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:50.173182 12631 tablet_bootstrap.cc:492] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:50.174038 12632 tablet_bootstrap.cc:492] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:50.174101 12631 tablet_bootstrap.cc:654] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:50.174161 12632 ts_tablet_manager.cc:1367] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:59:50.174676 12632 raft_consensus.cc:348] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:50.174777 12632 raft_consensus.cc:374] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:50.174805 12632 raft_consensus.cc:729] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:50.174957 12632 consensus_queue.cc:260] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:50.175334 12631 tablet_bootstrap.cc:492] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:50.175415 12631 ts_tablet_manager.cc:1367] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:50.175499 12632 ts_tablet_manager.cc:1398] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:50.175837 12631 raft_consensus.cc:348] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:50.175948 12631 raft_consensus.cc:374] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:50.175985 12631 raft_consensus.cc:729] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:50.176086 12632 tablet_bootstrap.cc:492] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:50.176110 12631 consensus_queue.cc:260] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:50.176375 12631 ts_tablet_manager.cc:1398] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:50.177294 12632 tablet_bootstrap.cc:654] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:50.179093 12632 tablet_bootstrap.cc:492] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:50.179200 12632 ts_tablet_manager.cc:1367] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:50.179183 12359 raft_consensus.cc:482] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:50.179262 12359 raft_consensus.cc:504] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:50.179664 12359 leader_election.cc:288] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:50.179667 12632 raft_consensus.cc:348] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:50.179771 12632 raft_consensus.cc:374] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:50.179807 12632 raft_consensus.cc:729] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:50.179929 11663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f11b5693d7456d8a604e6c8726e69a" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2" is_pre_election: true
I20240501 07:59:50.180024 11663 raft_consensus.cc:2454] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 0.
I20240501 07:59:50.180119 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f11b5693d7456d8a604e6c8726e69a" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6" is_pre_election: true
I20240501 07:59:50.180240 12632 consensus_queue.cc:260] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:50.180264 11531 raft_consensus.cc:2454] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 0.
I20240501 07:59:50.180338 11334 leader_election.cc:302] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, 655ebda1fb2c49c4af7087a760529835; no voters:
I20240501 07:59:50.180531 12632 ts_tablet_manager.cc:1398] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:50.180646 12359 raft_consensus.cc:2781] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:50.180783 12359 raft_consensus.cc:482] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:50.180876 12359 raft_consensus.cc:3037] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:50.181768 12359 raft_consensus.cc:504] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:50.182301 12359 leader_election.cc:288] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 election: Requested vote from peers 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:50.182739 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f11b5693d7456d8a604e6c8726e69a" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
I20240501 07:59:50.182870 11531 raft_consensus.cc:3037] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:50.183743 11531 raft_consensus.cc:2454] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 1.
I20240501 07:59:50.184037 11663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f11b5693d7456d8a604e6c8726e69a" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 07:59:50.184096 11336 leader_election.cc:302] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 655ebda1fb2c49c4af7087a760529835; no voters:
I20240501 07:59:50.184176 11663 raft_consensus.cc:3037] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:50.185076 11663 raft_consensus.cc:2454] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 1.
I20240501 07:59:50.185307 12359 raft_consensus.cc:2781] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:50.185530 12359 raft_consensus.cc:686] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Becoming Leader. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Running, Role: LEADER
I20240501 07:59:50.185708 12359 consensus_queue.cc:237] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:50.187466 11123 catalog_manager.cc:5484] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 reported cstate change: term changed from 0 to 1, leader changed from <none> to 655ebda1fb2c49c4af7087a760529835 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 07:59:50.188215 12535 raft_consensus.cc:482] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:50.188304 12535 raft_consensus.cc:504] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:50.188750 12535 leader_election.cc:288] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:50.189111 11270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21b6e2023e842739a7c0cc946e7c2fd" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" is_pre_election: true
I20240501 07:59:50.189270 11270 raft_consensus.cc:2454] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 0.
I20240501 07:59:50.189770 11597 leader_election.cc:302] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:50.189929 12535 raft_consensus.cc:2781] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:50.189985 12535 raft_consensus.cc:482] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:50.190021 12535 raft_consensus.cc:3037] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:50.190937 12535 raft_consensus.cc:504] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:50.191283 12535 leader_election.cc:288] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 election: Requested vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:50.191653 11270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21b6e2023e842739a7c0cc946e7c2fd" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 07:59:50.191774 11270 raft_consensus.cc:3037] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:50.192564 11270 raft_consensus.cc:2454] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 1.
I20240501 07:59:50.192838 11597 leader_election.cc:302] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:50.192979 12535 raft_consensus.cc:2781] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:50.193085 12535 raft_consensus.cc:686] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Becoming Leader. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Running, Role: LEADER
I20240501 07:59:50.193224 12535 consensus_queue.cc:237] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:50.193506 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21b6e2023e842739a7c0cc946e7c2fd" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835" is_pre_election: true
I20240501 07:59:50.193531 11400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21b6e2023e842739a7c0cc946e7c2fd" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 07:59:50.193639 11401 raft_consensus.cc:2454] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 0.
I20240501 07:59:50.195083 11123 catalog_manager.cc:5484] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 603919bb02364c04b6c530245b8b02f2 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 07:59:50.335790 12530 raft_consensus.cc:482] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:50.335938 12530 raft_consensus.cc:504] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:50.336409 12530 leader_election.cc:288] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:50.336861 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61e70ba02384f81b9ae36b1c65465b9" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835" is_pre_election: true
I20240501 07:59:50.336869 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61e70ba02384f81b9ae36b1c65465b9" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6" is_pre_election: true
I20240501 07:59:50.336997 11531 raft_consensus.cc:2454] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 0.
I20240501 07:59:50.337007 11401 raft_consensus.cc:2454] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 0.
I20240501 07:59:50.337302 11598 leader_election.cc:302] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 603919bb02364c04b6c530245b8b02f2; no voters:
I20240501 07:59:50.337471 12530 raft_consensus.cc:2781] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:50.337529 12530 raft_consensus.cc:482] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:50.337570 12530 raft_consensus.cc:3037] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:50.338495 12530 raft_consensus.cc:504] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:50.338845 12530 leader_election.cc:288] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 election: Requested vote from peers 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:50.339202 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61e70ba02384f81b9ae36b1c65465b9" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 07:59:50.339239 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61e70ba02384f81b9ae36b1c65465b9" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
I20240501 07:59:50.339315 11401 raft_consensus.cc:3037] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:50.339376 11531 raft_consensus.cc:3037] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:50.340085 11401 raft_consensus.cc:2454] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 1.
I20240501 07:59:50.340142 11531 raft_consensus.cc:2454] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 1.
I20240501 07:59:50.340409 11596 leader_election.cc:302] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, 655ebda1fb2c49c4af7087a760529835; no voters:
I20240501 07:59:50.340623 12530 raft_consensus.cc:2781] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:50.340732 12530 raft_consensus.cc:686] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Becoming Leader. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Running, Role: LEADER
I20240501 07:59:50.340878 12530 consensus_queue.cc:237] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:50.342718 11123 catalog_manager.cc:5484] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 603919bb02364c04b6c530245b8b02f2 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
W20240501 07:59:50.462817 12649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:50.463080 12649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:50.466320 12649 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 07:59:50.562446 11401 raft_consensus.cc:1261] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:50.562402 11531 raft_consensus.cc:1261] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:50.562891 12530 consensus_queue.cc:1035] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:50.563210 11531 raft_consensus.cc:1261] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:50.563616 12359 consensus_queue.cc:1035] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:50.564139 11270 raft_consensus.cc:1261] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:50.564631 11663 raft_consensus.cc:1261] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:50.564849 12535 consensus_queue.cc:1035] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:50.565124 11401 raft_consensus.cc:3037] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:50.565143 12535 consensus_queue.cc:1035] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:50.565873 12359 consensus_queue.cc:1035] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:50.566233 11401 raft_consensus.cc:1261] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:50.573230 12535 consensus_queue.cc:1035] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:50.583109 12610 mvcc.cc:205] Tried to move back new op lower bound from 7022798399745085440 to 7022798398232891392. Current Snapshot: MvccSnapshot[applied={T|T < 7022798399745085440}]
I20240501 07:59:50.601629 12539 mvcc.cc:205] Tried to move back new op lower bound from 7022798399777595392 to 7022798398202114048. Current Snapshot: MvccSnapshot[applied={T|T < 7022798399777595392 or (T in {7022798399777595392})}]
INSERT report
rows total: 2000
time total: 110.233 ms
time per row: 0.0551164 ms
I20240501 07:59:50.626791 11124 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49370:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:50.632789 11380 tablet_service.cc:1467] Processing CreateTablet for tablet 164fd265561e47128e7ff585b8cfab07 (DEFAULT_TABLE table=rebalance_test_table_6 [id=701e50fb54354219a26221f76a7352d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:50.632761 11248 tablet_service.cc:1467] Processing CreateTablet for tablet 164fd265561e47128e7ff585b8cfab07 (DEFAULT_TABLE table=rebalance_test_table_6 [id=701e50fb54354219a26221f76a7352d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:50.632949 11511 tablet_service.cc:1467] Processing CreateTablet for tablet 164fd265561e47128e7ff585b8cfab07 (DEFAULT_TABLE table=rebalance_test_table_6 [id=701e50fb54354219a26221f76a7352d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:50.633095 11248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164fd265561e47128e7ff585b8cfab07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.633091 11380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164fd265561e47128e7ff585b8cfab07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.633247 11511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164fd265561e47128e7ff585b8cfab07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.633917 11512 tablet_service.cc:1467] Processing CreateTablet for tablet 5b7de08d2f004cd8abc2bbe1cf7b1444 (DEFAULT_TABLE table=rebalance_test_table_6 [id=701e50fb54354219a26221f76a7352d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:50.633968 11250 tablet_service.cc:1467] Processing CreateTablet for tablet 5b7de08d2f004cd8abc2bbe1cf7b1444 (DEFAULT_TABLE table=rebalance_test_table_6 [id=701e50fb54354219a26221f76a7352d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:50.634163 11512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7de08d2f004cd8abc2bbe1cf7b1444. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.634215 11250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7de08d2f004cd8abc2bbe1cf7b1444. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.634303 11510 tablet_service.cc:1467] Processing CreateTablet for tablet ea97f34590684bcf998228245dbcd9b0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=701e50fb54354219a26221f76a7352d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:50.634558 11510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea97f34590684bcf998228245dbcd9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.635043 11249 tablet_service.cc:1467] Processing CreateTablet for tablet ea97f34590684bcf998228245dbcd9b0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=701e50fb54354219a26221f76a7352d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:50.635293 11249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea97f34590684bcf998228245dbcd9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.635346 12631 tablet_bootstrap.cc:492] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:50.636050 11380 tablet_service.cc:1467] Processing CreateTablet for tablet ea97f34590684bcf998228245dbcd9b0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=701e50fb54354219a26221f76a7352d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:50.636050 11642 tablet_service.cc:1467] Processing CreateTablet for tablet 5b7de08d2f004cd8abc2bbe1cf7b1444 (DEFAULT_TABLE table=rebalance_test_table_6 [id=701e50fb54354219a26221f76a7352d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:50.636282 11380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea97f34590684bcf998228245dbcd9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.636301 11642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7de08d2f004cd8abc2bbe1cf7b1444. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:50.636307 12631 tablet_bootstrap.cc:654] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:50.637274 12631 tablet_bootstrap.cc:492] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:50.637365 12631 ts_tablet_manager.cc:1367] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:50.637854 12631 raft_consensus.cc:348] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:50.637965 12631 raft_consensus.cc:374] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:50.638011 12631 raft_consensus.cc:729] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:50.638131 12631 consensus_queue.cc:260] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:50.638278 12632 tablet_bootstrap.cc:492] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:50.638442 12631 ts_tablet_manager.cc:1398] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:50.638621 12631 tablet_bootstrap.cc:492] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:50.639204 12632 tablet_bootstrap.cc:654] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:50.639397 12631 tablet_bootstrap.cc:654] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:50.640298 12632 tablet_bootstrap.cc:492] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:50.640390 12632 ts_tablet_manager.cc:1367] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:50.640786 12632 raft_consensus.cc:348] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:50.640889 12632 raft_consensus.cc:374] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:50.640909 12632 raft_consensus.cc:729] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:50.641026 12632 consensus_queue.cc:260] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:50.641278 12632 ts_tablet_manager.cc:1398] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:50.642127 12631 tablet_bootstrap.cc:492] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:50.642211 12631 ts_tablet_manager.cc:1367] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:50.642678 12631 raft_consensus.cc:348] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:50.642800 12631 raft_consensus.cc:374] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:50.642835 12631 raft_consensus.cc:729] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:50.643002 12631 consensus_queue.cc:260] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:50.643292 12631 ts_tablet_manager.cc:1398] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:50.652690 12637 tablet_bootstrap.cc:492] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:50.652690 12633 tablet_bootstrap.cc:492] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:50.653650 12633 tablet_bootstrap.cc:654] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:50.653650 12637 tablet_bootstrap.cc:654] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:50.654793 12633 tablet_bootstrap.cc:492] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:50.654887 12633 ts_tablet_manager.cc:1367] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 07:59:50.654963 12637 tablet_bootstrap.cc:492] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:50.655061 12637 ts_tablet_manager.cc:1367] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:50.655284 12633 raft_consensus.cc:348] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:50.655383 12633 raft_consensus.cc:374] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:50.655413 12633 raft_consensus.cc:729] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:50.655411 12637 raft_consensus.cc:348] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:50.655522 12637 raft_consensus.cc:374] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:50.655555 12637 raft_consensus.cc:729] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:50.655540 12633 consensus_queue.cc:260] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:50.655678 12637 consensus_queue.cc:260] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:50.655820 12633 ts_tablet_manager.cc:1398] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:50.655938 12637 ts_tablet_manager.cc:1398] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:50.655980 12633 tablet_bootstrap.cc:492] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:50.656102 12637 tablet_bootstrap.cc:492] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:50.657029 12637 tablet_bootstrap.cc:654] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:50.657084 12633 tablet_bootstrap.cc:654] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:50.658123 12633 tablet_bootstrap.cc:492] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:50.658205 12633 ts_tablet_manager.cc:1367] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:50.658241 12637 tablet_bootstrap.cc:492] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:50.658317 12637 ts_tablet_manager.cc:1367] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 07:59:50.658685 12633 raft_consensus.cc:348] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:50.658798 12633 raft_consensus.cc:374] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:50.658828 12633 raft_consensus.cc:729] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:50.658799 12637 raft_consensus.cc:348] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:50.658910 12637 raft_consensus.cc:374] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:50.658939 12637 raft_consensus.cc:729] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:50.658943 12633 consensus_queue.cc:260] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:50.659049 12637 consensus_queue.cc:260] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:50.659229 12633 ts_tablet_manager.cc:1398] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:50.659323 12637 ts_tablet_manager.cc:1398] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:50.659389 12633 tablet_bootstrap.cc:492] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:50.659472 12637 tablet_bootstrap.cc:492] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:50.660324 12633 tablet_bootstrap.cc:654] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:50.660405 12637 tablet_bootstrap.cc:654] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:50.662581 12637 tablet_bootstrap.cc:492] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:50.662667 12637 ts_tablet_manager.cc:1367] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:50.663045 12633 tablet_bootstrap.cc:492] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:50.663126 12633 ts_tablet_manager.cc:1367] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 07:59:50.663092 12637 raft_consensus.cc:348] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:50.663204 12637 raft_consensus.cc:374] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:50.663232 12637 raft_consensus.cc:729] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:50.663344 12637 consensus_queue.cc:260] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:50.663549 12633 raft_consensus.cc:348] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:50.663641 12637 ts_tablet_manager.cc:1398] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:50.663651 12633 raft_consensus.cc:374] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:50.663681 12633 raft_consensus.cc:729] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:50.663805 12633 consensus_queue.cc:260] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:50.664139 12633 ts_tablet_manager.cc:1398] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:50.700392 12676 raft_consensus.cc:482] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:50.700542 12676 raft_consensus.cc:504] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:50.700985 12676 leader_election.cc:288] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:50.701517 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7de08d2f004cd8abc2bbe1cf7b1444" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6" is_pre_election: true
I20240501 07:59:50.701598 11270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7de08d2f004cd8abc2bbe1cf7b1444" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" is_pre_election: true
I20240501 07:59:50.701700 11531 raft_consensus.cc:2454] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 0.
I20240501 07:59:50.701781 11270 raft_consensus.cc:2454] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 0.
I20240501 07:59:50.702047 11598 leader_election.cc:302] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 603919bb02364c04b6c530245b8b02f2; no voters:
I20240501 07:59:50.702239 12676 raft_consensus.cc:2781] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:50.702294 12676 raft_consensus.cc:482] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:50.702363 12676 raft_consensus.cc:3037] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:50.703280 12676 raft_consensus.cc:504] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:50.703694 12676 leader_election.cc:288] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 election: Requested vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:50.704073 11270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7de08d2f004cd8abc2bbe1cf7b1444" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 07:59:50.704133 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7de08d2f004cd8abc2bbe1cf7b1444" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
I20240501 07:59:50.704191 11270 raft_consensus.cc:3037] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:50.704259 11531 raft_consensus.cc:3037] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:50.704967 11270 raft_consensus.cc:2454] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 1.
I20240501 07:59:50.705085 11531 raft_consensus.cc:2454] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 1.
I20240501 07:59:50.705277 11597 leader_election.cc:302] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:50.705514 12676 raft_consensus.cc:2781] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:50.705590 12676 raft_consensus.cc:686] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Becoming Leader. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Running, Role: LEADER
I20240501 07:59:50.705718 12676 consensus_queue.cc:237] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:50.707631 11121 catalog_manager.cc:5484] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 603919bb02364c04b6c530245b8b02f2 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 07:59:50.732776 12644 raft_consensus.cc:482] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:50.732921 12644 raft_consensus.cc:504] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:50.733376 12644 leader_election.cc:288] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:50.733866 11270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164fd265561e47128e7ff585b8cfab07" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" is_pre_election: true
I20240501 07:59:50.733873 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164fd265561e47128e7ff585b8cfab07" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6" is_pre_election: true
I20240501 07:59:50.734000 11270 raft_consensus.cc:2454] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 0.
I20240501 07:59:50.734019 11531 raft_consensus.cc:2454] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 0.
I20240501 07:59:50.734349 11335 leader_election.cc:302] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:50.734584 12644 raft_consensus.cc:2781] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:50.734649 12644 raft_consensus.cc:482] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:50.734683 12644 raft_consensus.cc:3037] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:50.735759 12644 raft_consensus.cc:504] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:50.736143 12644 leader_election.cc:288] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 election: Requested vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:50.736486 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164fd265561e47128e7ff585b8cfab07" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
I20240501 07:59:50.736486 11270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164fd265561e47128e7ff585b8cfab07" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 07:59:50.736619 11270 raft_consensus.cc:3037] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:50.736619 11531 raft_consensus.cc:3037] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:50.737643 11270 raft_consensus.cc:2454] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 1.
I20240501 07:59:50.737645 11531 raft_consensus.cc:2454] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 1.
I20240501 07:59:50.737973 11335 leader_election.cc:302] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:50.738184 12644 raft_consensus.cc:2781] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:50.738284 12644 raft_consensus.cc:686] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Becoming Leader. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Running, Role: LEADER
I20240501 07:59:50.738495 12644 consensus_queue.cc:237] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:50.740290 11121 catalog_manager.cc:5484] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: term changed from 0 to 1, leader changed from <none> to 655ebda1fb2c49c4af7087a760529835 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 07:59:50.952335 12442 raft_consensus.cc:482] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:50.952476 12442 raft_consensus.cc:504] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:50.952904 12442 leader_election.cc:288] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:50.953364 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea97f34590684bcf998228245dbcd9b0" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6" is_pre_election: true
I20240501 07:59:50.953403 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea97f34590684bcf998228245dbcd9b0" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835" is_pre_election: true
I20240501 07:59:50.953492 11531 raft_consensus.cc:2454] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 0.
I20240501 07:59:50.953581 11401 raft_consensus.cc:2454] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 0.
I20240501 07:59:50.953814 11205 leader_election.cc:302] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:50.954000 12442 raft_consensus.cc:2781] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:50.954062 12442 raft_consensus.cc:482] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:50.954087 12442 raft_consensus.cc:3037] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:50.954946 12442 raft_consensus.cc:504] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:50.955289 12442 leader_election.cc:288] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 election: Requested vote from peers 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:50.955693 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea97f34590684bcf998228245dbcd9b0" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 07:59:50.955789 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea97f34590684bcf998228245dbcd9b0" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
I20240501 07:59:50.955824 11401 raft_consensus.cc:3037] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:50.955910 11531 raft_consensus.cc:3037] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:50.956645 11401 raft_consensus.cc:2454] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 1.
I20240501 07:59:50.956737 11531 raft_consensus.cc:2454] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 1.
I20240501 07:59:50.957026 11203 leader_election.cc:302] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:50.957267 12442 raft_consensus.cc:2781] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:50.957414 12442 raft_consensus.cc:686] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Becoming Leader. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Running, Role: LEADER
I20240501 07:59:50.957571 12442 consensus_queue.cc:237] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:50.959321 11121 catalog_manager.cc:5484] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 07:59:51.112372 12676 consensus_queue.cc:1035] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:51.116050 12676 consensus_queue.cc:1035] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 07:59:51.123149 12699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:51.123426 12699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:51.126799 12699 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 07:59:51.211158 11270 raft_consensus.cc:1261] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:51.211742 11531 raft_consensus.cc:1261] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:51.212210 11401 raft_consensus.cc:1261] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:51.212608 12644 consensus_queue.cc:1035] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:51.212651 12442 consensus_queue.cc:1035] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:51.212896 12644 consensus_queue.cc:1035] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:51.219164 11530 raft_consensus.cc:1261] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:51.220093 12617 consensus_queue.cc:1035] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:51.227383 12702 mvcc.cc:205] Tried to move back new op lower bound from 7022798402401386496 to 7022798401363533824. Current Snapshot: MvccSnapshot[applied={T|T < 7022798402401386496}]
INSERT report
rows total: 2000
time total: 114.826 ms
time per row: 0.057413 ms
I20240501 07:59:51.289976 11122 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49370:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:51.296638 11380 tablet_service.cc:1467] Processing CreateTablet for tablet 53193305f868433c852755e8a8a81766 (DEFAULT_TABLE table=rebalance_test_table_7 [id=36e20074d8dc4402bd7d9b2039213af5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:51.297119 11380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53193305f868433c852755e8a8a81766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.297050 11642 tablet_service.cc:1467] Processing CreateTablet for tablet 53193305f868433c852755e8a8a81766 (DEFAULT_TABLE table=rebalance_test_table_7 [id=36e20074d8dc4402bd7d9b2039213af5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:51.297397 11642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53193305f868433c852755e8a8a81766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.297442 11641 tablet_service.cc:1467] Processing CreateTablet for tablet 6d214be86534458091f54a386d7d3334 (DEFAULT_TABLE table=rebalance_test_table_7 [id=36e20074d8dc4402bd7d9b2039213af5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:51.297667 11641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d214be86534458091f54a386d7d3334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.299726 11643 tablet_service.cc:1467] Processing CreateTablet for tablet 5e8550ea95984d08937b9b3e2af12bec (DEFAULT_TABLE table=rebalance_test_table_7 [id=36e20074d8dc4402bd7d9b2039213af5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:51.299811 11249 tablet_service.cc:1467] Processing CreateTablet for tablet 53193305f868433c852755e8a8a81766 (DEFAULT_TABLE table=rebalance_test_table_7 [id=36e20074d8dc4402bd7d9b2039213af5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:51.299958 11643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8550ea95984d08937b9b3e2af12bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.300087 11249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53193305f868433c852755e8a8a81766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.300082 11510 tablet_service.cc:1467] Processing CreateTablet for tablet 6d214be86534458091f54a386d7d3334 (DEFAULT_TABLE table=rebalance_test_table_7 [id=36e20074d8dc4402bd7d9b2039213af5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:51.300323 11510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d214be86534458091f54a386d7d3334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.302563 11512 tablet_service.cc:1467] Processing CreateTablet for tablet 5e8550ea95984d08937b9b3e2af12bec (DEFAULT_TABLE table=rebalance_test_table_7 [id=36e20074d8dc4402bd7d9b2039213af5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:51.302829 11512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8550ea95984d08937b9b3e2af12bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.302781 11380 tablet_service.cc:1467] Processing CreateTablet for tablet 5e8550ea95984d08937b9b3e2af12bec (DEFAULT_TABLE table=rebalance_test_table_7 [id=36e20074d8dc4402bd7d9b2039213af5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:51.303020 11380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8550ea95984d08937b9b3e2af12bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.303864 12759 tablet_bootstrap.cc:492] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:51.305331 12759 tablet_bootstrap.cc:654] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:51.305442 11249 tablet_service.cc:1467] Processing CreateTablet for tablet 6d214be86534458091f54a386d7d3334 (DEFAULT_TABLE table=rebalance_test_table_7 [id=36e20074d8dc4402bd7d9b2039213af5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:51.305737 11249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d214be86534458091f54a386d7d3334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.307174 12760 tablet_bootstrap.cc:492] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:51.308174 12760 tablet_bootstrap.cc:654] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:51.308423 12762 tablet_bootstrap.cc:492] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:51.309268 12759 tablet_bootstrap.cc:492] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:51.309324 12762 tablet_bootstrap.cc:654] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:51.309377 12759 ts_tablet_manager.cc:1367] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:51.310042 12761 tablet_bootstrap.cc:492] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:51.309978 12759 raft_consensus.cc:348] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:51.310102 12759 raft_consensus.cc:374] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:51.310132 12759 raft_consensus.cc:729] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:51.310259 12759 consensus_queue.cc:260] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:51.310573 12759 ts_tablet_manager.cc:1398] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:51.310842 12759 tablet_bootstrap.cc:492] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:51.310976 12761 tablet_bootstrap.cc:654] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:51.311765 12759 tablet_bootstrap.cc:654] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:51.311985 12762 tablet_bootstrap.cc:492] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:51.312089 12762 ts_tablet_manager.cc:1367] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:51.312603 12762 raft_consensus.cc:348] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:51.312721 12762 raft_consensus.cc:374] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:51.312752 12762 raft_consensus.cc:729] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:51.312883 12762 consensus_queue.cc:260] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:51.313069 12760 tablet_bootstrap.cc:492] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:51.313158 12760 ts_tablet_manager.cc:1367] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:51.313164 12762 ts_tablet_manager.cc:1398] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:51.313324 12762 tablet_bootstrap.cc:492] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:51.313585 12760 raft_consensus.cc:348] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:51.313694 12760 raft_consensus.cc:374] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:51.313722 12760 raft_consensus.cc:729] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:51.313863 12760 consensus_queue.cc:260] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:51.314221 12762 tablet_bootstrap.cc:654] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:51.315047 12760 ts_tablet_manager.cc:1398] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:51.315227 12760 tablet_bootstrap.cc:492] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:51.315237 12761 tablet_bootstrap.cc:492] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:51.315335 12761 ts_tablet_manager.cc:1367] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:51.315784 12761 raft_consensus.cc:348] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:51.315901 12761 raft_consensus.cc:374] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:51.315932 12761 raft_consensus.cc:729] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:51.316072 12760 tablet_bootstrap.cc:654] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:51.316447 12762 tablet_bootstrap.cc:492] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:51.316535 12762 ts_tablet_manager.cc:1367] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:51.317167 12762 raft_consensus.cc:348] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:51.317322 12762 raft_consensus.cc:374] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:51.317366 12762 raft_consensus.cc:729] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:51.317512 12762 consensus_queue.cc:260] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:51.317834 12762 ts_tablet_manager.cc:1398] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:51.317991 12759 tablet_bootstrap.cc:492] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:51.318110 12759 ts_tablet_manager.cc:1367] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 07:59:51.318080 12761 consensus_queue.cc:260] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:51.318665 12759 raft_consensus.cc:348] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:51.318778 12759 raft_consensus.cc:374] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:51.318812 12759 raft_consensus.cc:729] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:51.319113 12761 ts_tablet_manager.cc:1398] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:51.319141 12759 consensus_queue.cc:260] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:51.319322 12761 tablet_bootstrap.cc:492] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:51.319528 12759 ts_tablet_manager.cc:1398] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:51.320468 12760 tablet_bootstrap.cc:492] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:51.320598 12760 ts_tablet_manager.cc:1367] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:59:51.321010 12761 tablet_bootstrap.cc:654] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:51.321123 12760 raft_consensus.cc:348] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:51.321251 12760 raft_consensus.cc:374] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:51.321282 12760 raft_consensus.cc:729] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:51.321415 12760 consensus_queue.cc:260] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:51.321780 12760 ts_tablet_manager.cc:1398] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:51.321983 12760 tablet_bootstrap.cc:492] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:51.322517 12761 tablet_bootstrap.cc:492] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:51.322614 12761 ts_tablet_manager.cc:1367] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:51.323057 12760 tablet_bootstrap.cc:654] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:51.323139 12761 raft_consensus.cc:348] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:51.323249 12761 raft_consensus.cc:374] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:51.323285 12761 raft_consensus.cc:729] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:51.323412 12761 consensus_queue.cc:260] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:51.323720 12761 ts_tablet_manager.cc:1398] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:51.324210 12760 tablet_bootstrap.cc:492] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:51.324306 12760 ts_tablet_manager.cc:1367] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:51.324807 12760 raft_consensus.cc:348] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:51.324926 12760 raft_consensus.cc:374] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:51.324960 12760 raft_consensus.cc:729] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:51.325086 12760 consensus_queue.cc:260] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:51.325419 12760 ts_tablet_manager.cc:1398] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:51.379601 12359 raft_consensus.cc:482] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:51.379745 12359 raft_consensus.cc:504] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:51.380152 12359 leader_election.cc:288] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:51.380668 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8550ea95984d08937b9b3e2af12bec" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6" is_pre_election: true
I20240501 07:59:51.380698 11663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8550ea95984d08937b9b3e2af12bec" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2" is_pre_election: true
I20240501 07:59:51.380812 11531 raft_consensus.cc:2454] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 0.
I20240501 07:59:51.380841 11663 raft_consensus.cc:2454] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 0.
I20240501 07:59:51.381172 11336 leader_election.cc:302] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 655ebda1fb2c49c4af7087a760529835; no voters:
I20240501 07:59:51.381350 12359 raft_consensus.cc:2781] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:51.381412 12359 raft_consensus.cc:482] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:51.381438 12359 raft_consensus.cc:3037] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:51.382329 12359 raft_consensus.cc:504] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:51.382766 12359 leader_election.cc:288] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 election: Requested vote from peers 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:51.383145 11663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8550ea95984d08937b9b3e2af12bec" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 07:59:51.383152 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8550ea95984d08937b9b3e2af12bec" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
I20240501 07:59:51.383263 11531 raft_consensus.cc:3037] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:51.383267 11663 raft_consensus.cc:3037] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:51.384100 11663 raft_consensus.cc:2454] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 1.
I20240501 07:59:51.384124 11531 raft_consensus.cc:2454] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 1.
I20240501 07:59:51.384409 11334 leader_election.cc:302] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, 655ebda1fb2c49c4af7087a760529835; no voters:
I20240501 07:59:51.384588 12359 raft_consensus.cc:2781] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:51.384681 12359 raft_consensus.cc:686] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Becoming Leader. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Running, Role: LEADER
I20240501 07:59:51.384824 12359 consensus_queue.cc:237] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:51.386662 11126 catalog_manager.cc:5484] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 reported cstate change: term changed from 0 to 1, leader changed from <none> to 655ebda1fb2c49c4af7087a760529835 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 07:59:51.396570 12738 raft_consensus.cc:482] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:51.396720 12738 raft_consensus.cc:504] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:51.397200 12738 leader_election.cc:288] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 07:59:51.397717 11663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d214be86534458091f54a386d7d3334" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2" is_pre_election: true
I20240501 07:59:51.397702 11269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d214be86534458091f54a386d7d3334" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" is_pre_election: true
I20240501 07:59:51.397867 11663 raft_consensus.cc:2454] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 0.
I20240501 07:59:51.397867 11269 raft_consensus.cc:2454] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 0.
I20240501 07:59:51.398208 11465 leader_election.cc:302] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 603919bb02364c04b6c530245b8b02f2; no voters:
I20240501 07:59:51.398403 12738 raft_consensus.cc:2781] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:51.398500 12738 raft_consensus.cc:482] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:51.398538 12738 raft_consensus.cc:3037] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:51.399381 12738 raft_consensus.cc:504] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:51.399750 12738 leader_election.cc:288] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 election: Requested vote from peers 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 07:59:51.400177 11269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d214be86534458091f54a386d7d3334" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 07:59:51.400177 11663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d214be86534458091f54a386d7d3334" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 07:59:51.400316 11663 raft_consensus.cc:3037] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:51.400316 11269 raft_consensus.cc:3037] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:51.401297 11663 raft_consensus.cc:2454] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 1.
I20240501 07:59:51.401296 11269 raft_consensus.cc:2454] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 1.
I20240501 07:59:51.401618 11466 leader_election.cc:302] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:51.401809 12738 raft_consensus.cc:2781] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:51.401888 12738 raft_consensus.cc:686] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Becoming Leader. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Running, Role: LEADER
I20240501 07:59:51.402047 12738 consensus_queue.cc:237] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 07:59:51.404037 11126 catalog_manager.cc:5484] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 07:59:51.411583 12748 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:51.411754 12748 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:51.412199 12748 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:51.412732 11269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53193305f868433c852755e8a8a81766" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" is_pre_election: true
I20240501 07:59:51.412732 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53193305f868433c852755e8a8a81766" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835" is_pre_election: true
I20240501 07:59:51.412892 11269 raft_consensus.cc:2454] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 0.
I20240501 07:59:51.412892 11401 raft_consensus.cc:2454] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 0.
I20240501 07:59:51.413208 11597 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:51.413395 12748 raft_consensus.cc:2781] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:51.413456 12748 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:51.413481 12748 raft_consensus.cc:3037] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:51.414341 12748 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:51.414741 12748 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 election: Requested vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:51.415146 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53193305f868433c852755e8a8a81766" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 07:59:51.415163 11269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53193305f868433c852755e8a8a81766" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 07:59:51.415266 11269 raft_consensus.cc:3037] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:51.415270 11401 raft_consensus.cc:3037] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:51.416041 11401 raft_consensus.cc:2454] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 1.
I20240501 07:59:51.416047 11269 raft_consensus.cc:2454] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 1.
I20240501 07:59:51.416365 11597 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:51.416558 12748 raft_consensus.cc:2781] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:51.416647 12748 raft_consensus.cc:686] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Becoming Leader. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Running, Role: LEADER
I20240501 07:59:51.416806 12748 consensus_queue.cc:237] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:51.418699 11126 catalog_manager.cc:5484] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 603919bb02364c04b6c530245b8b02f2 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
W20240501 07:59:51.540339 12775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:51.540611 12775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:51.543988 12775 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 07:59:51.630821 11663 raft_consensus.cc:1261] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:51.630961 11269 raft_consensus.cc:1261] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:51.631446 11401 raft_consensus.cc:1261] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:51.631422 12738 consensus_queue.cc:1035] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:51.631788 12738 consensus_queue.cc:1035] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:51.632263 12748 consensus_queue.cc:1035] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:51.632766 11531 raft_consensus.cc:1261] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:51.633174 12359 consensus_queue.cc:1035] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:51.634395 11662 raft_consensus.cc:1261] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:51.635969 12644 consensus_queue.cc:1035] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:51.643514 11270 raft_consensus.cc:1261] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:51.644280 12676 consensus_queue.cc:1035] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 120.816 ms
time per row: 0.0604081 ms
I20240501 07:59:51.712234 11122 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49370:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:51.718036 11249 tablet_service.cc:1467] Processing CreateTablet for tablet b4d15e19eed54aefb6b96276fad98324 (DEFAULT_TABLE table=rebalance_test_table_8 [id=df5f9864bbf8499dbaa392bcb85701c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:51.718036 11380 tablet_service.cc:1467] Processing CreateTablet for tablet b4d15e19eed54aefb6b96276fad98324 (DEFAULT_TABLE table=rebalance_test_table_8 [id=df5f9864bbf8499dbaa392bcb85701c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:51.718353 11380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4d15e19eed54aefb6b96276fad98324. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.718353 11249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4d15e19eed54aefb6b96276fad98324. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.718323 11248 tablet_service.cc:1467] Processing CreateTablet for tablet 5c47aaeb16c945ae84ece505aada2a7c (DEFAULT_TABLE table=rebalance_test_table_8 [id=df5f9864bbf8499dbaa392bcb85701c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:51.718631 11248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c47aaeb16c945ae84ece505aada2a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.719101 11512 tablet_service.cc:1467] Processing CreateTablet for tablet b4d15e19eed54aefb6b96276fad98324 (DEFAULT_TABLE table=rebalance_test_table_8 [id=df5f9864bbf8499dbaa392bcb85701c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:51.719352 11512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4d15e19eed54aefb6b96276fad98324. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.720577 12761 tablet_bootstrap.cc:492] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:51.720710 11510 tablet_service.cc:1467] Processing CreateTablet for tablet ed88ea311f474ab28cc6fbe6069258bf (DEFAULT_TABLE table=rebalance_test_table_8 [id=df5f9864bbf8499dbaa392bcb85701c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:51.720925 11510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed88ea311f474ab28cc6fbe6069258bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.721361 12761 tablet_bootstrap.cc:654] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:51.721407 12762 tablet_bootstrap.cc:492] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:51.722201 12762 tablet_bootstrap.cc:654] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:51.722404 11643 tablet_service.cc:1467] Processing CreateTablet for tablet 5c47aaeb16c945ae84ece505aada2a7c (DEFAULT_TABLE table=rebalance_test_table_8 [id=df5f9864bbf8499dbaa392bcb85701c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:51.722520 11642 tablet_service.cc:1467] Processing CreateTablet for tablet ed88ea311f474ab28cc6fbe6069258bf (DEFAULT_TABLE table=rebalance_test_table_8 [id=df5f9864bbf8499dbaa392bcb85701c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:51.722671 11643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c47aaeb16c945ae84ece505aada2a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.722947 12759 tablet_bootstrap.cc:492] T b4d15e19eed54aefb6b96276fad98324 P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:51.723873 12759 tablet_bootstrap.cc:654] T b4d15e19eed54aefb6b96276fad98324 P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:51.724216 11642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed88ea311f474ab28cc6fbe6069258bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.724817 12760 tablet_bootstrap.cc:492] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:51.725013 12759 tablet_bootstrap.cc:492] T b4d15e19eed54aefb6b96276fad98324 P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:51.725093 12759 ts_tablet_manager.cc:1367] T b4d15e19eed54aefb6b96276fad98324 P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:51.725582 12759 raft_consensus.cc:348] T b4d15e19eed54aefb6b96276fad98324 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:51.725690 12759 raft_consensus.cc:374] T b4d15e19eed54aefb6b96276fad98324 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:51.725720 12759 raft_consensus.cc:729] T b4d15e19eed54aefb6b96276fad98324 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:51.725745 12760 tablet_bootstrap.cc:654] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:51.725852 12759 consensus_queue.cc:260] T b4d15e19eed54aefb6b96276fad98324 P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:51.726135 12759 ts_tablet_manager.cc:1398] T b4d15e19eed54aefb6b96276fad98324 P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:51.726989 12762 tablet_bootstrap.cc:492] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:51.727080 12762 ts_tablet_manager.cc:1367] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240501 07:59:51.727205 11248 tablet_service.cc:1467] Processing CreateTablet for tablet ed88ea311f474ab28cc6fbe6069258bf (DEFAULT_TABLE table=rebalance_test_table_8 [id=df5f9864bbf8499dbaa392bcb85701c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:51.727480 11248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed88ea311f474ab28cc6fbe6069258bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.727586 12762 raft_consensus.cc:348] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:51.727728 12762 raft_consensus.cc:374] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:51.727769 12762 raft_consensus.cc:729] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:51.727918 12762 consensus_queue.cc:260] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:51.728201 12762 ts_tablet_manager.cc:1398] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:51.728374 12762 tablet_bootstrap.cc:492] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:51.728549 11380 tablet_service.cc:1467] Processing CreateTablet for tablet 5c47aaeb16c945ae84ece505aada2a7c (DEFAULT_TABLE table=rebalance_test_table_8 [id=df5f9864bbf8499dbaa392bcb85701c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:51.728956 11380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c47aaeb16c945ae84ece505aada2a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:51.729302 12762 tablet_bootstrap.cc:654] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:51.731331 12760 tablet_bootstrap.cc:492] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:51.731458 12760 ts_tablet_manager.cc:1367] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 07:59:51.731561 12759 tablet_bootstrap.cc:492] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:51.731844 12762 tablet_bootstrap.cc:492] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:51.731920 12762 ts_tablet_manager.cc:1367] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 07:59:51.731958 12760 raft_consensus.cc:348] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:51.732084 12760 raft_consensus.cc:374] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:51.732111 12760 raft_consensus.cc:729] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:51.732225 12760 consensus_queue.cc:260] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:51.732348 12762 raft_consensus.cc:348] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:51.732471 12762 raft_consensus.cc:374] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:51.732492 12760 ts_tablet_manager.cc:1398] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:51.732510 12762 raft_consensus.cc:729] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:51.732535 12759 tablet_bootstrap.cc:654] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:51.732631 12762 consensus_queue.cc:260] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:51.732908 12762 ts_tablet_manager.cc:1398] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:51.735265 12759 tablet_bootstrap.cc:492] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:51.735368 12759 ts_tablet_manager.cc:1367] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:51.735702 12761 tablet_bootstrap.cc:492] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:51.735808 12761 ts_tablet_manager.cc:1367] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20240501 07:59:51.735805 12759 raft_consensus.cc:348] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:51.735931 12759 raft_consensus.cc:374] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:51.735965 12759 raft_consensus.cc:729] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:51.736099 12759 consensus_queue.cc:260] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:51.736263 12761 raft_consensus.cc:348] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:51.736377 12761 raft_consensus.cc:374] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:51.736383 12759 ts_tablet_manager.cc:1398] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:51.736413 12761 raft_consensus.cc:729] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:51.736531 12761 consensus_queue.cc:260] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:51.736909 12761 ts_tablet_manager.cc:1398] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:51.737712 12761 tablet_bootstrap.cc:492] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:51.738778 12760 tablet_bootstrap.cc:492] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:51.739725 12760 tablet_bootstrap.cc:654] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:51.740705 12761 tablet_bootstrap.cc:654] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:51.741577 12760 tablet_bootstrap.cc:492] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:51.741672 12760 ts_tablet_manager.cc:1367] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 07:59:51.742069 12760 raft_consensus.cc:348] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:51.742182 12760 raft_consensus.cc:374] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:51.742210 12760 raft_consensus.cc:729] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:51.742341 12760 consensus_queue.cc:260] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:51.742839 12761 tablet_bootstrap.cc:492] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:51.742933 12761 ts_tablet_manager.cc:1367] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:51.743402 12761 raft_consensus.cc:348] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:51.743500 12761 raft_consensus.cc:374] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:51.743530 12761 raft_consensus.cc:729] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:51.743851 12761 consensus_queue.cc:260] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:51.744217 12760 ts_tablet_manager.cc:1398] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:51.744459 12761 ts_tablet_manager.cc:1398] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:51.745088 12761 tablet_bootstrap.cc:492] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:51.746201 12761 tablet_bootstrap.cc:654] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:51.747884 12761 tablet_bootstrap.cc:492] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:51.747979 12761 ts_tablet_manager.cc:1367] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:51.748524 12761 raft_consensus.cc:348] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:51.748631 12761 raft_consensus.cc:374] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:51.748652 12761 raft_consensus.cc:729] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:51.748759 12761 consensus_queue.cc:260] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:51.749037 12761 ts_tablet_manager.cc:1398] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:51.833114 12644 raft_consensus.cc:482] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:51.833266 12644 raft_consensus.cc:504] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:51.833695 12644 leader_election.cc:288] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:51.834203 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c47aaeb16c945ae84ece505aada2a7c" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2" is_pre_election: true
I20240501 07:59:51.834201 11268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c47aaeb16c945ae84ece505aada2a7c" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" is_pre_election: true
I20240501 07:59:51.834352 11268 raft_consensus.cc:2454] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 0.
I20240501 07:59:51.834354 11662 raft_consensus.cc:2454] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 0.
I20240501 07:59:51.834707 11334 leader_election.cc:302] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, 655ebda1fb2c49c4af7087a760529835; no voters:
I20240501 07:59:51.834901 12644 raft_consensus.cc:2781] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:51.834966 12644 raft_consensus.cc:482] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:51.835012 12644 raft_consensus.cc:3037] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:51.835973 12644 raft_consensus.cc:504] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:51.836350 12644 leader_election.cc:288] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 election: Requested vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:51.836866 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c47aaeb16c945ae84ece505aada2a7c" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 07:59:51.836866 11268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c47aaeb16c945ae84ece505aada2a7c" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 07:59:51.837023 11662 raft_consensus.cc:3037] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:51.837030 11268 raft_consensus.cc:3037] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:51.838044 11268 raft_consensus.cc:2454] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 1.
I20240501 07:59:51.838044 11662 raft_consensus.cc:2454] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 1.
I20240501 07:59:51.838481 11335 leader_election.cc:302] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, 655ebda1fb2c49c4af7087a760529835, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:51.838697 12644 raft_consensus.cc:2781] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:51.838812 12644 raft_consensus.cc:686] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Becoming Leader. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Running, Role: LEADER
I20240501 07:59:51.838963 12644 consensus_queue.cc:237] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:51.840855 11123 catalog_manager.cc:5484] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 reported cstate change: term changed from 0 to 1, leader changed from <none> to 655ebda1fb2c49c4af7087a760529835 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 07:59:51.877672 12748 raft_consensus.cc:482] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:51.877799 12748 raft_consensus.cc:504] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:51.878151 12748 leader_election.cc:288] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:51.878567 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed88ea311f474ab28cc6fbe6069258bf" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6" is_pre_election: true
I20240501 07:59:51.878599 11268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed88ea311f474ab28cc6fbe6069258bf" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" is_pre_election: true
I20240501 07:59:51.878721 11531 raft_consensus.cc:2454] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 0.
I20240501 07:59:51.878721 11268 raft_consensus.cc:2454] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 0.
I20240501 07:59:51.879017 11597 leader_election.cc:302] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 603919bb02364c04b6c530245b8b02f2, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:51.879194 12748 raft_consensus.cc:2781] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:51.879251 12748 raft_consensus.cc:482] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:51.879297 12748 raft_consensus.cc:3037] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:51.880100 12748 raft_consensus.cc:504] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:51.880447 12748 leader_election.cc:288] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 election: Requested vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:51.880779 11268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed88ea311f474ab28cc6fbe6069258bf" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 07:59:51.880828 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed88ea311f474ab28cc6fbe6069258bf" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
I20240501 07:59:51.880918 11268 raft_consensus.cc:3037] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:51.880931 11531 raft_consensus.cc:3037] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:51.881853 11531 raft_consensus.cc:2454] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 1.
I20240501 07:59:51.881853 11268 raft_consensus.cc:2454] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 1.
I20240501 07:59:51.882174 11598 leader_election.cc:302] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 603919bb02364c04b6c530245b8b02f2; no voters:
I20240501 07:59:51.882356 12748 raft_consensus.cc:2781] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:51.882476 12748 raft_consensus.cc:686] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Becoming Leader. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Running, Role: LEADER
I20240501 07:59:51.882615 12748 consensus_queue.cc:237] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 07:59:51.884430 11123 catalog_manager.cc:5484] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 603919bb02364c04b6c530245b8b02f2 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 07:59:51.964797 12524 raft_consensus.cc:482] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:51.964952 12524 raft_consensus.cc:504] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:51.965386 12524 leader_election.cc:288] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:51.965776 11268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d15e19eed54aefb6b96276fad98324" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" is_pre_election: true
I20240501 07:59:51.965893 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d15e19eed54aefb6b96276fad98324" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835" is_pre_election: true
I20240501 07:59:51.965939 11268 raft_consensus.cc:2454] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 0.
I20240501 07:59:51.966028 11401 raft_consensus.cc:2454] T b4d15e19eed54aefb6b96276fad98324 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 0.
I20240501 07:59:51.966219 11466 leader_election.cc:302] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:51.966410 12524 raft_consensus.cc:2781] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:51.966490 12524 raft_consensus.cc:482] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:51.966534 12524 raft_consensus.cc:3037] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:51.967324 12524 raft_consensus.cc:504] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:51.967666 12524 leader_election.cc:288] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 election: Requested vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:51.968032 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d15e19eed54aefb6b96276fad98324" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 07:59:51.968176 11401 raft_consensus.cc:3037] T b4d15e19eed54aefb6b96276fad98324 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:51.968168 11268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d15e19eed54aefb6b96276fad98324" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 07:59:51.968303 11268 raft_consensus.cc:3037] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:51.969172 11401 raft_consensus.cc:2454] T b4d15e19eed54aefb6b96276fad98324 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 1.
I20240501 07:59:51.969170 11268 raft_consensus.cc:2454] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 1.
I20240501 07:59:51.969476 11465 leader_election.cc:302] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 655ebda1fb2c49c4af7087a760529835; no voters:
I20240501 07:59:51.969739 12524 raft_consensus.cc:2781] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:51.969821 12524 raft_consensus.cc:686] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Becoming Leader. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Running, Role: LEADER
I20240501 07:59:51.969954 12524 consensus_queue.cc:237] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:51.971859 11123 catalog_manager.cc:5484] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
W20240501 07:59:52.145138 12831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:52.145455 12831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:52.148842 12831 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 07:59:52.249245 11401 raft_consensus.cc:1261] T b4d15e19eed54aefb6b96276fad98324 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:52.249296 11268 raft_consensus.cc:1261] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:52.249698 12524 consensus_queue.cc:1035] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:52.249925 12738 consensus_queue.cc:1035] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:52.252883 11531 raft_consensus.cc:1261] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:52.253301 12748 consensus_queue.cc:1035] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:52.254848 11269 raft_consensus.cc:1261] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:52.255236 12748 consensus_queue.cc:1035] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:52.257344 11662 raft_consensus.cc:1261] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:52.257740 12644 consensus_queue.cc:1035] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:52.258970 11270 raft_consensus.cc:1261] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:52.259366 12644 consensus_queue.cc:1035] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 134.392 ms
time per row: 0.067196 ms
I20240501 07:59:52.329034 11122 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49370:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:52.335018 11642 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6e4ae97e884219ac7d0d94997d9cc9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1deda0b4866b43c9ba7c69dae5d65e4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:52.335050 11380 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6e4ae97e884219ac7d0d94997d9cc9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1deda0b4866b43c9ba7c69dae5d65e4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:52.335361 11642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6e4ae97e884219ac7d0d94997d9cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:52.335363 11380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6e4ae97e884219ac7d0d94997d9cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:52.337735 12900 tablet_bootstrap.cc:492] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:52.337735 12899 tablet_bootstrap.cc:492] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:52.338219 11510 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6e4ae97e884219ac7d0d94997d9cc9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1deda0b4866b43c9ba7c69dae5d65e4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:52.338560 11511 tablet_service.cc:1467] Processing CreateTablet for tablet 18b146cc4e164e1d9f9f827217c9b3ab (DEFAULT_TABLE table=rebalance_test_table_9 [id=1deda0b4866b43c9ba7c69dae5d65e4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:52.338776 12900 tablet_bootstrap.cc:654] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:52.338819 11511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b146cc4e164e1d9f9f827217c9b3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:52.339035 12899 tablet_bootstrap.cc:654] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:52.339468 11510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6e4ae97e884219ac7d0d94997d9cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:52.338200 11512 tablet_service.cc:1467] Processing CreateTablet for tablet 7c23e3bd7ec34876956046e5d32e4d82 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1deda0b4866b43c9ba7c69dae5d65e4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:52.339815 11512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c23e3bd7ec34876956046e5d32e4d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:52.340325 12899 tablet_bootstrap.cc:492] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:52.340472 12899 ts_tablet_manager.cc:1367] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:52.341029 12899 raft_consensus.cc:348] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:52.341300 12899 raft_consensus.cc:374] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:52.341390 12899 raft_consensus.cc:729] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:52.341593 12899 consensus_queue.cc:260] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:52.341873 11642 tablet_service.cc:1467] Processing CreateTablet for tablet 7c23e3bd7ec34876956046e5d32e4d82 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1deda0b4866b43c9ba7c69dae5d65e4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:52.342218 11381 tablet_service.cc:1467] Processing CreateTablet for tablet 18b146cc4e164e1d9f9f827217c9b3ab (DEFAULT_TABLE table=rebalance_test_table_9 [id=1deda0b4866b43c9ba7c69dae5d65e4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:52.342311 11380 tablet_service.cc:1467] Processing CreateTablet for tablet 7c23e3bd7ec34876956046e5d32e4d82 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1deda0b4866b43c9ba7c69dae5d65e4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:52.342526 11381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b146cc4e164e1d9f9f827217c9b3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:52.342658 11380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c23e3bd7ec34876956046e5d32e4d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:52.343299 11642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c23e3bd7ec34876956046e5d32e4d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:52.341991 12899 ts_tablet_manager.cc:1398] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:52.345119 12899 tablet_bootstrap.cc:492] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 07:59:52.345280 11248 tablet_service.cc:1467] Processing CreateTablet for tablet 18b146cc4e164e1d9f9f827217c9b3ab (DEFAULT_TABLE table=rebalance_test_table_9 [id=1deda0b4866b43c9ba7c69dae5d65e4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:52.345578 11248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b146cc4e164e1d9f9f827217c9b3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:52.346172 12899 tablet_bootstrap.cc:654] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:52.346796 12900 tablet_bootstrap.cc:492] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:52.346889 12900 ts_tablet_manager.cc:1367] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:52.347357 12900 raft_consensus.cc:348] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:52.347473 12900 raft_consensus.cc:374] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:52.347503 12900 raft_consensus.cc:729] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:52.347631 12900 consensus_queue.cc:260] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:52.347750 12899 tablet_bootstrap.cc:492] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: No bootstrap required, opened a new log
I20240501 07:59:52.347828 12899 ts_tablet_manager.cc:1367] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:52.347903 12900 ts_tablet_manager.cc:1398] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:52.348067 12900 tablet_bootstrap.cc:492] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:52.348260 12899 raft_consensus.cc:348] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:52.348361 12899 raft_consensus.cc:374] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:52.348387 12899 raft_consensus.cc:729] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 07:59:52.348520 12899 consensus_queue.cc:260] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:52.348779 12899 ts_tablet_manager.cc:1398] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:52.348965 12900 tablet_bootstrap.cc:654] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:52.349059 12903 tablet_bootstrap.cc:492] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:52.349941 12903 tablet_bootstrap.cc:654] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:52.350626 12905 tablet_bootstrap.cc:492] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 07:59:52.350972 12900 tablet_bootstrap.cc:492] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:52.351073 12900 ts_tablet_manager.cc:1367] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:52.351382 12905 tablet_bootstrap.cc:654] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:52.351545 12900 raft_consensus.cc:348] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:52.351722 12900 raft_consensus.cc:374] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:52.351780 12900 raft_consensus.cc:729] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:52.351902 12900 consensus_queue.cc:260] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:52.352245 12900 ts_tablet_manager.cc:1398] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:52.352411 12905 tablet_bootstrap.cc:492] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05: No bootstrap required, opened a new log
I20240501 07:59:52.352450 12900 tablet_bootstrap.cc:492] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 07:59:52.352502 12905 ts_tablet_manager.cc:1367] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:52.352902 12905 raft_consensus.cc:348] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:52.352996 12905 raft_consensus.cc:374] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:52.353049 12905 raft_consensus.cc:729] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 07:59:52.353188 12905 consensus_queue.cc:260] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:52.353803 12900 tablet_bootstrap.cc:654] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:52.353904 12905 ts_tablet_manager.cc:1398] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:52.354485 12903 tablet_bootstrap.cc:492] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:52.354635 12903 ts_tablet_manager.cc:1367] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 07:59:52.355044 12903 raft_consensus.cc:348] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:52.355134 12903 raft_consensus.cc:374] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:52.355173 12903 raft_consensus.cc:729] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:52.355485 12900 tablet_bootstrap.cc:492] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835: No bootstrap required, opened a new log
I20240501 07:59:52.355556 12900 ts_tablet_manager.cc:1367] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 07:59:52.355716 12903 consensus_queue.cc:260] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:52.355954 12900 raft_consensus.cc:348] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:52.356058 12900 raft_consensus.cc:374] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:52.356068 12903 ts_tablet_manager.cc:1398] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:52.356120 12900 raft_consensus.cc:729] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 07:59:52.356240 12903 tablet_bootstrap.cc:492] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:52.356259 12900 consensus_queue.cc:260] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:52.356541 12900 ts_tablet_manager.cc:1398] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:52.357275 12903 tablet_bootstrap.cc:654] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:52.358403 12903 tablet_bootstrap.cc:492] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:52.358516 12903 ts_tablet_manager.cc:1367] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 07:59:52.358887 12903 raft_consensus.cc:348] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:52.358975 12903 raft_consensus.cc:374] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:52.358995 12903 raft_consensus.cc:729] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:52.359103 12903 consensus_queue.cc:260] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:52.359371 12903 ts_tablet_manager.cc:1398] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:52.359534 12903 tablet_bootstrap.cc:492] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 07:59:52.360410 12903 tablet_bootstrap.cc:654] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:52.362035 12903 tablet_bootstrap.cc:492] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6: No bootstrap required, opened a new log
I20240501 07:59:52.362131 12903 ts_tablet_manager.cc:1367] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:59:52.362649 12903 raft_consensus.cc:348] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:52.362763 12903 raft_consensus.cc:374] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:52.362797 12903 raft_consensus.cc:729] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 07:59:52.362892 12903 consensus_queue.cc:260] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:52.363178 12903 ts_tablet_manager.cc:1398] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:52.468783 12896 raft_consensus.cc:482] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:52.468928 12896 raft_consensus.cc:504] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:52.469347 12896 leader_election.cc:288] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:52.469816 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6e4ae97e884219ac7d0d94997d9cc9" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6" is_pre_election: true
I20240501 07:59:52.469816 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6e4ae97e884219ac7d0d94997d9cc9" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2" is_pre_election: true
I20240501 07:59:52.469959 11531 raft_consensus.cc:2454] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 0.
I20240501 07:59:52.469959 11662 raft_consensus.cc:2454] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 0.
I20240501 07:59:52.470293 11334 leader_election.cc:302] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, 655ebda1fb2c49c4af7087a760529835; no voters:
I20240501 07:59:52.470578 12896 raft_consensus.cc:2781] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:52.470650 12896 raft_consensus.cc:482] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:52.470686 12896 raft_consensus.cc:3037] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:52.471524 12896 raft_consensus.cc:504] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:52.471869 12896 leader_election.cc:288] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 election: Requested vote from peers 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:52.472226 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6e4ae97e884219ac7d0d94997d9cc9" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 07:59:52.472234 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6e4ae97e884219ac7d0d94997d9cc9" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
I20240501 07:59:52.472347 11662 raft_consensus.cc:3037] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:52.472363 11531 raft_consensus.cc:3037] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:52.473088 11662 raft_consensus.cc:2454] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 1.
I20240501 07:59:52.473251 11531 raft_consensus.cc:2454] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 1.
I20240501 07:59:52.473383 11334 leader_election.cc:302] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, 655ebda1fb2c49c4af7087a760529835; no voters:
I20240501 07:59:52.473548 12896 raft_consensus.cc:2781] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:52.473627 12896 raft_consensus.cc:686] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Becoming Leader. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Running, Role: LEADER
I20240501 07:59:52.473763 12896 consensus_queue.cc:237] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 07:59:52.475953 11122 catalog_manager.cc:5484] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: term changed from 0 to 1, leader changed from <none> to 655ebda1fb2c49c4af7087a760529835 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 07:59:52.497563 12885 raft_consensus.cc:482] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:52.497709 12885 raft_consensus.cc:504] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:52.498179 12885 leader_election.cc:288] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:52.498652 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b146cc4e164e1d9f9f827217c9b3ab" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6" is_pre_election: true
I20240501 07:59:52.498651 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b146cc4e164e1d9f9f827217c9b3ab" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835" is_pre_election: true
I20240501 07:59:52.498790 11401 raft_consensus.cc:2454] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 0.
I20240501 07:59:52.498793 11531 raft_consensus.cc:2454] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 0.
I20240501 07:59:52.499086 11205 leader_election.cc:302] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:52.499251 12885 raft_consensus.cc:2781] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:52.499310 12885 raft_consensus.cc:482] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:52.499336 12885 raft_consensus.cc:3037] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:52.500314 12885 raft_consensus.cc:504] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:52.500679 12885 leader_election.cc:288] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 election: Requested vote from peers 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:52.500993 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b146cc4e164e1d9f9f827217c9b3ab" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
I20240501 07:59:52.501044 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18b146cc4e164e1d9f9f827217c9b3ab" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 07:59:52.501111 11531 raft_consensus.cc:3037] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:52.501175 11401 raft_consensus.cc:3037] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:52.501911 11531 raft_consensus.cc:2454] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 1.
I20240501 07:59:52.502039 11401 raft_consensus.cc:2454] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 1.
I20240501 07:59:52.502230 11205 leader_election.cc:302] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 07:59:52.502393 12885 raft_consensus.cc:2781] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:52.502509 12885 raft_consensus.cc:686] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Becoming Leader. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Running, Role: LEADER
I20240501 07:59:52.502878 12885 consensus_queue.cc:237] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:52.504817 11122 catalog_manager.cc:5484] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 07:59:52.614173 12896 raft_consensus.cc:482] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:52.614324 12896 raft_consensus.cc:504] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:52.614779 12896 leader_election.cc:288] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:52.615305 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c23e3bd7ec34876956046e5d32e4d82" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2" is_pre_election: true
I20240501 07:59:52.615305 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c23e3bd7ec34876956046e5d32e4d82" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6" is_pre_election: true
I20240501 07:59:52.615463 11662 raft_consensus.cc:2454] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 0.
I20240501 07:59:52.615463 11531 raft_consensus.cc:2454] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 0.
I20240501 07:59:52.615806 11336 leader_election.cc:302] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 603919bb02364c04b6c530245b8b02f2, 655ebda1fb2c49c4af7087a760529835; no voters:
I20240501 07:59:52.615993 12896 raft_consensus.cc:2781] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:52.616052 12896 raft_consensus.cc:482] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:52.616086 12896 raft_consensus.cc:3037] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:52.616850 12896 raft_consensus.cc:504] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:52.617203 12896 leader_election.cc:288] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 election: Requested vote from peers 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:52.617698 11531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c23e3bd7ec34876956046e5d32e4d82" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
I20240501 07:59:52.617699 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c23e3bd7ec34876956046e5d32e4d82" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 07:59:52.617818 11531 raft_consensus.cc:3037] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:52.617828 11662 raft_consensus.cc:3037] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:52.618613 11531 raft_consensus.cc:2454] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 1.
I20240501 07:59:52.618613 11662 raft_consensus.cc:2454] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 1.
I20240501 07:59:52.618932 11336 leader_election.cc:302] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 655ebda1fb2c49c4af7087a760529835; no voters:
I20240501 07:59:52.619097 12896 raft_consensus.cc:2781] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:52.619194 12896 raft_consensus.cc:686] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Becoming Leader. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Running, Role: LEADER
I20240501 07:59:52.619347 12896 consensus_queue.cc:237] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 07:59:52.621106 11122 catalog_manager.cc:5484] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: term changed from 0 to 1, leader changed from <none> to 655ebda1fb2c49c4af7087a760529835 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
W20240501 07:59:52.756775 12915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:52.757061 12915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:52.760404 12915 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 07:59:52.851799 11662 raft_consensus.cc:1261] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:52.851872 11531 raft_consensus.cc:1261] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:52.852330 12815 consensus_queue.cc:1035] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:52.852661 12896 consensus_queue.cc:1035] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:52.857652 11530 raft_consensus.cc:1261] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:52.857847 11663 raft_consensus.cc:1261] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:52.858150 12896 consensus_queue.cc:1035] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:52.858439 12815 consensus_queue.cc:1035] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:52.878206 12946 mvcc.cc:205] Tried to move back new op lower bound from 7022798409177468928 to 7022798407573966848. Current Snapshot: MvccSnapshot[applied={T|T < 7022798409177468928 or (T in {7022798409177468928})}]
I20240501 07:59:52.889415 12963 mvcc.cc:205] Tried to move back new op lower bound from 7022798409117048832 to 7022798408170250240. Current Snapshot: MvccSnapshot[applied={T|T < 7022798409117048832}]
I20240501 07:59:52.893764 11531 raft_consensus.cc:1261] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:52.893762 11401 raft_consensus.cc:1261] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:52.894253 12442 consensus_queue.cc:1035] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:52.894407 12885 consensus_queue.cc:1035] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 123.239 ms
time per row: 0.0616197 ms
I20240501 07:59:52.930092 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:42805
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:41001
--webserver_port=44695
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 07:59:53.040452 12972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:53.040721 12972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:53.040786 12972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:53.044278 12972 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:53.044374 12972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:53.044467 12972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:59:53.048538 12972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:42805
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=44695
--tserver_master_addrs=127.0.99.190:41001
--never_fsync=true
--heap_profile_path=/tmp/kudu.12972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:53.049510 12972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:53.050834 12972 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:53.059772 12978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:53.060163 12980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:53.060678 12977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:53.061205 12972 server_base.cc:1034] running on GCE node
I20240501 07:59:53.061619 12972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:53.062243 12972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:53.063427 12972 hybrid_clock.cc:648] HybridClock initialized: now 1714550393063417 us; error 55 us; skew 500 ppm
I20240501 07:59:53.065771 12972 webserver.cc:460] Webserver started at http://127.0.99.133:44695/ using document root <none> and password file <none>
I20240501 07:59:53.066500 12972 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:53.066588 12972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:53.070215 12972 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:53.072335 12986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:53.073513 12972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:53.073637 12972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "60e8857abe7c41038af46ebb5a095a90"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:53.074074 12972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:53.085332 12972 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:53.086041 12972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:53.086211 12972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:53.086830 12972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:53.087922 12972 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:53.087977 12972 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:53.088025 12972 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:53.088057 12972 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:53.097745 12972 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:42805
I20240501 07:59:53.097796 13099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:42805 every 8 connection(s)
I20240501 07:59:53.098841 12972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:59:53.106081 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 12972
I20240501 07:59:53.106616 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:36771
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:41001
--webserver_port=42815
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:53.108667 13100 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41001
I20240501 07:59:53.108924 13100 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:53.109491 13100 heartbeater.cc:511] Master 127.0.99.190:41001 requested a full tablet report, sending...
I20240501 07:59:53.110806 11127 ts_manager.cc:194] Re-registered known tserver with Master: 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
I20240501 07:59:53.111580 11127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:44825
W20240501 07:59:53.221275 13103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:53.221516 13103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:53.221561 13103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:53.225132 13103 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:53.225214 13103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:53.225297 13103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 07:59:53.229771 13103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:36771
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=42815
--tserver_master_addrs=127.0.99.190:41001
--never_fsync=true
--heap_profile_path=/tmp/kudu.13103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:53.230747 13103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:53.231843 13103 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:53.240561 13112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:53.241094 13109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:53.241663 13103 server_base.cc:1034] running on GCE node
W20240501 07:59:53.242070 13110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:53.242537 13103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:53.243221 13103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:53.244385 13103 hybrid_clock.cc:648] HybridClock initialized: now 1714550393244358 us; error 45 us; skew 500 ppm
I20240501 07:59:53.246749 13103 webserver.cc:460] Webserver started at http://127.0.99.134:42815/ using document root <none> and password file <none>
I20240501 07:59:53.247468 13103 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:53.247570 13103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:53.251490 13103 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20240501 07:59:53.253557 13118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:53.254645 13103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:53.254771 13103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "53d1d2c0767b4f6195b69758a3dd6cf9"
format_stamp: "Formatted at 2024-05-01 07:59:42 on dist-test-slave-2srn"
I20240501 07:59:53.255196 13103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:53.273175 13103 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:53.273840 13103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:53.274001 13103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:53.274591 13103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:53.275488 13103 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:53.275538 13103 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:53.275590 13103 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:53.275614 13103 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:53.285039 13103 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:36771
I20240501 07:59:53.285159 13231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:36771 every 8 connection(s)
I20240501 07:59:53.286144 13103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 07:59:53.293393 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 13103
I20240501 07:59:53.293870 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:43995
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:41001
--webserver_port=37353
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:53.296506 13232 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41001
I20240501 07:59:53.296775 13232 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:53.297305 13232 heartbeater.cc:511] Master 127.0.99.190:41001 requested a full tablet report, sending...
I20240501 07:59:53.298458 11127 ts_manager.cc:194] Re-registered known tserver with Master: 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771)
I20240501 07:59:53.299088 11127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:50339
W20240501 07:59:53.416863 13235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:53.417109 13235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:53.417166 13235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:53.420668 13235 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:53.420753 13235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:53.420841 13235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 07:59:53.424950 13235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:43995
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=37353
--tserver_master_addrs=127.0.99.190:41001
--never_fsync=true
--heap_profile_path=/tmp/kudu.13235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:53.425912 13235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:53.427078 13235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:53.435696 13240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:53.436129 13243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:53.436758 13241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:53.437186 13235 server_base.cc:1034] running on GCE node
I20240501 07:59:53.437601 13235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:53.438278 13235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:53.439484 13235 hybrid_clock.cc:648] HybridClock initialized: now 1714550393439458 us; error 50 us; skew 500 ppm
I20240501 07:59:53.445384 13235 webserver.cc:460] Webserver started at http://127.0.99.135:37353/ using document root <none> and password file <none>
I20240501 07:59:53.445976 13235 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:53.446050 13235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:53.449776 13235 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:53.451959 13249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:53.453199 13235 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:53.453323 13235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "e5c03261d31c4d9a91077a17e1cd0336"
format_stamp: "Formatted at 2024-05-01 07:59:43 on dist-test-slave-2srn"
I20240501 07:59:53.453848 13235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:53.475416 13235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:53.476040 13235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:53.476205 13235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:53.476779 13235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:53.477685 13235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:53.477735 13235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:53.477789 13235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:53.477815 13235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:53.487097 13235 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:43995
I20240501 07:59:53.487125 13362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:43995 every 8 connection(s)
I20240501 07:59:53.488163 13235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 07:59:53.495471 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 13235
I20240501 07:59:53.496068 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
/tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:34503
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:41001
--webserver_port=36501
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 07:59:53.497620 13363 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41001
I20240501 07:59:53.497865 13363 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:53.498350 13363 heartbeater.cc:511] Master 127.0.99.190:41001 requested a full tablet report, sending...
I20240501 07:59:53.499405 11127 ts_manager.cc:194] Re-registered known tserver with Master: e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
I20240501 07:59:53.499944 11127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:38429
W20240501 07:59:53.610713 13366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:53.610966 13366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:53.611028 13366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:53.614251 13366 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 07:59:53.614351 13366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:53.614485 13366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240501 07:59:53.618468 13366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:34503
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=36501
--tserver_master_addrs=127.0.99.190:41001
--never_fsync=true
--heap_profile_path=/tmp/kudu.13366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:53.619654 13366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:53.620848 13366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:53.629871 13371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:53.630292 13372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:53.630836 13374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:53.631356 13366 server_base.cc:1034] running on GCE node
I20240501 07:59:53.631837 13366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:53.632489 13366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:53.633648 13366 hybrid_clock.cc:648] HybridClock initialized: now 1714550393633631 us; error 38 us; skew 500 ppm
I20240501 07:59:53.635751 13366 webserver.cc:460] Webserver started at http://127.0.99.136:36501/ using document root <none> and password file <none>
I20240501 07:59:53.636322 13366 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:53.636402 13366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:53.640067 13366 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20240501 07:59:53.642048 13380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:53.643213 13366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 07:59:53.643323 13366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "ca177138f2c44d2eafa1646b795f7608"
format_stamp: "Formatted at 2024-05-01 07:59:43 on dist-test-slave-2srn"
I20240501 07:59:53.643750 13366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:53.666056 13366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:53.666766 13366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:53.666922 13366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:53.667495 13366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:53.668577 13366 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:53.668634 13366 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:53.668700 13366 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:53.668721 13366 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:53.678544 13366 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:34503
I20240501 07:59:53.678581 13494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:34503 every 8 connection(s)
I20240501 07:59:53.679618 13366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 07:59:53.682510 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu as pid 13366
I20240501 07:59:53.692234 13495 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41001
I20240501 07:59:53.692525 13495 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:53.693150 13495 heartbeater.cc:511] Master 127.0.99.190:41001 requested a full tablet report, sending...
I20240501 07:59:53.694229 11127 ts_manager.cc:194] Re-registered known tserver with Master: ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503)
I20240501 07:59:53.694994 11127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:54267
I20240501 07:59:54.097553 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.098266 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.107443 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.109979 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:54.114167 13100 heartbeater.cc:503] Master 127.0.99.190:41001 was elected leader, sending a full tablet report...
I20240501 07:59:54.119951 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.134156 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.146977 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.154009 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.168946 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.174062 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:54.194284 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.203706 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.205935 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.214393 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.223640 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.223851 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:54.224704 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:54.231195 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:54.248059 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.278002 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:54.287871 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.301895 13232 heartbeater.cc:503] Master 127.0.99.190:41001 was elected leader, sending a full tablet report...
I20240501 07:59:54.307786 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.312634 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.314020 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.325716 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.328622 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.340451 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.345773 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.349278 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.359334 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:54.364034 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:54.390619 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.409538 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.409811 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.420996 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.437211 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.444013 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:54.444206 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.453193 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.461467 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:54.502313 13363 heartbeater.cc:503] Master 127.0.99.190:41001 was elected leader, sending a full tablet report...
I20240501 07:59:54.630534 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.632501 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.666484 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.669008 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.680855 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:54.684890 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.697685 13495 heartbeater.cc:503] Master 127.0.99.190:41001 was elected leader, sending a full tablet report...
I20240501 07:59:54.701598 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:54.706508 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.707123 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.708237 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.726462 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.727044 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:54.732612 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.741050 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:54.747433 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.779186 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.807667 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:54.819150 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.824298 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.830610 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.846508 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.850833 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.854252 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:54.873271 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.874671 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.884598 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.887091 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.899587 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.903226 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:54.908280 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:54.912940 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.913617 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.913817 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:54.914041 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.917284 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.946640 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:54.967347 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.983399 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.987455 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:54.993813 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:55.159514 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:55.163561 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.172734 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.174930 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:55.179425 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.180809 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:55.181499 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.187103 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.203416 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.207703 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.219882 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:55.222733 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:55.230751 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:55.231669 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:55.251148 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.251960 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:55.377293 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.378639 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.385905 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.387100 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:55.388693 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:55.397856 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:55.402175 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:55.428265 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.433763 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:55.442157 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.444309 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:55.447818 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:55.450779 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.466583 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.475109 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:55.490681 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:55.496169 11269 consensus_queue.cc:237] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:55.498981 11401 raft_consensus.cc:1261] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:55.499647 13728 consensus_queue.cc:1035] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:55.503877 11663 raft_consensus.cc:1261] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:55.506348 13728 consensus_queue.cc:1035] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:55.510190 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.524281 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.530645 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:55.535771 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.537514 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.543509 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:55.557029 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.565797 14240 raft_consensus.cc:2932] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:55.571918 11663 raft_consensus.cc:2932] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:55.572479 11122 catalog_manager.cc:5484] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:55.574723 11401 raft_consensus.cc:2932] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:55.578586 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 07:59:55.595842 11206 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2e63d24d7df4fa3b2b26695e70ebe9f. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:55.610211 11270 consensus_queue.cc:237] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:55.611989 11663 raft_consensus.cc:1261] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:55.612460 14241 consensus_queue.cc:1035] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:55.613443 11531 raft_consensus.cc:1261] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:55.614014 14241 consensus_queue.cc:1035] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:55.617465 11206 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd8348990f1b496bb62d8c0484193442. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:55.621192 13728 raft_consensus.cc:2932] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:55.624758 11663 raft_consensus.cc:2932] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:55.625242 11119 catalog_manager.cc:5484] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:55.625160 11531 raft_consensus.cc:2932] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:55.665007 11531 consensus_queue.cc:237] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:55.667644 11401 raft_consensus.cc:1261] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:55.668067 11663 raft_consensus.cc:1261] T 93602521de174997bda6702d25aafce1 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:55.668947 13107 consensus_queue.cc:1035] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:55.669404 13107 consensus_queue.cc:1035] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:55.673741 11465 consensus_peers.cc:485] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771): Couldn't send request to peer 53d1d2c0767b4f6195b69758a3dd6cf9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 93602521de174997bda6702d25aafce1. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:55.676354 14244 raft_consensus.cc:2932] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:55.679661 11401 raft_consensus.cc:2932] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:55.682317 11122 catalog_manager.cc:5484] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:55.682906 11663 raft_consensus.cc:2932] T 93602521de174997bda6702d25aafce1 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:55.703660 11270 consensus_queue.cc:237] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:55.705814 11401 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:55.706156 11663 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:55.706449 14289 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:55.706827 14290 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:55.710970 11203 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771): Couldn't send request to peer 53d1d2c0767b4f6195b69758a3dd6cf9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25cee7e51c7c438aa721d71fb82dc71e. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:55.714601 14290 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:55.716316 11663 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:55.719308 11401 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:55.721374 11127 catalog_manager.cc:5484] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 reported cstate change: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:55.742344 11401 consensus_queue.cc:237] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:55.744208 11663 raft_consensus.cc:1261] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:55.745049 14068 consensus_queue.cc:1035] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:55.745179 11531 raft_consensus.cc:1261] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:55.745994 14068 consensus_queue.cc:1035] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:55.763914 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64613ea537c842f29d810e829278394e. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:55.765980 11531 raft_consensus.cc:2932] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:55.768497 14068 raft_consensus.cc:2932] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:55.771332 11663 raft_consensus.cc:2932] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:55.774924 11119 catalog_manager.cc:5484] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2 reported cstate change: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:55.814353 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.819092 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:55.828994 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:55.830642 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.836880 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.837190 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.838822 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:55.844192 11270 consensus_queue.cc:237] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
W20240501 07:59:55.847570 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18b146cc4e164e1d9f9f827217c9b3ab. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:55.847748 11531 raft_consensus.cc:1261] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:55.848342 14290 consensus_queue.cc:1035] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:55.853487 11401 raft_consensus.cc:1261] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:55.854025 14290 consensus_queue.cc:1035] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:55.857710 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:55.865873 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.888448 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:55.888648 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.890293 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:55.890291 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:55.894572 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.920538 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.936748 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:55.938864 14290 raft_consensus.cc:2932] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:55.941545 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:55.944159 11401 raft_consensus.cc:2932] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:55.944552 11125 catalog_manager.cc:5484] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:55.955204 11531 raft_consensus.cc:2932] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:55.959210 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.970199 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:55.980957 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:55.989408 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:55.993569 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:56.002810 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:56.008407 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:56.036749 14540 ts_tablet_manager.cc:902] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608: Initiating tablet copy from peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 07:59:56.037729 14540 tablet_copy_client.cc:308] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46085
I20240501 07:59:56.048374 11290 tablet_copy_service.cc:140] P e2dafc1866314669ad0de8fb7bf7fd05: Received BeginTabletCopySession request for tablet f2e63d24d7df4fa3b2b26695e70ebe9f from peer ca177138f2c44d2eafa1646b795f7608 ({username='slave'} at 127.0.99.136:55185)
I20240501 07:59:56.048574 11290 tablet_copy_service.cc:161] P e2dafc1866314669ad0de8fb7bf7fd05: Beginning new tablet copy session on tablet f2e63d24d7df4fa3b2b26695e70ebe9f from peer ca177138f2c44d2eafa1646b795f7608 at {username='slave'} at 127.0.99.136:55185: session id = ca177138f2c44d2eafa1646b795f7608-f2e63d24d7df4fa3b2b26695e70ebe9f
I20240501 07:59:56.048981 11531 consensus_queue.cc:237] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:56.050199 11290 tablet_copy_source_session.cc:205] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:56.051054 11270 raft_consensus.cc:1261] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.051713 13107 consensus_queue.cc:1035] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.052024 11662 raft_consensus.cc:1261] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.052467 13107 consensus_queue.cc:1035] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.054358 14540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2e63d24d7df4fa3b2b26695e70ebe9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:56.055804 11401 consensus_queue.cc:237] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:56.057646 11662 raft_consensus.cc:1261] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 07:59:56.057760 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64613ea537c842f29d810e829278394e. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:56.058797 11531 raft_consensus.cc:1261] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:56.059314 14457 consensus_queue.cc:1035] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 07:59:56.062135 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d214be86534458091f54a386d7d3334. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:56.062536 14457 consensus_queue.cc:1035] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 07:59:56.064517 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64613ea537c842f29d810e829278394e. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:56.064590 14456 raft_consensus.cc:2932] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.065083 11531 raft_consensus.cc:2932] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.067934 11120 catalog_manager.cc:5484] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 4 to 5, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.068879 11662 raft_consensus.cc:2932] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.075836 14560 ts_tablet_manager.cc:902] T 93602521de174997bda6702d25aafce1 P 53d1d2c0767b4f6195b69758a3dd6cf9: Initiating tablet copy from peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:56.076666 14560 tablet_copy_client.cc:308] T 93602521de174997bda6702d25aafce1 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37957
I20240501 07:59:56.077363 14540 tablet_copy_client.cc:783] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:56.077777 14540 tablet_copy_client.cc:647] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:56.079978 14540 tablet_copy_client.cc:523] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:56.082051 11552 tablet_copy_service.cc:140] P 22ec12774a4047a8b859743dc73f8ff6: Received BeginTabletCopySession request for tablet 93602521de174997bda6702d25aafce1 from peer 53d1d2c0767b4f6195b69758a3dd6cf9 ({username='slave'} at 127.0.99.134:40967)
I20240501 07:59:56.082211 11552 tablet_copy_service.cc:161] P 22ec12774a4047a8b859743dc73f8ff6: Beginning new tablet copy session on tablet 93602521de174997bda6702d25aafce1 from peer 53d1d2c0767b4f6195b69758a3dd6cf9 at {username='slave'} at 127.0.99.134:40967: session id = 53d1d2c0767b4f6195b69758a3dd6cf9-93602521de174997bda6702d25aafce1
I20240501 07:59:56.083763 11552 tablet_copy_source_session.cc:205] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:56.084357 14540 tablet_bootstrap.cc:492] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608: Bootstrap starting.
I20240501 07:59:56.088950 14560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93602521de174997bda6702d25aafce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:56.095232 13107 raft_consensus.cc:2932] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.097105 14560 tablet_copy_client.cc:783] T 93602521de174997bda6702d25aafce1 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:56.097169 11270 raft_consensus.cc:2932] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.097342 14560 tablet_copy_client.cc:647] T 93602521de174997bda6702d25aafce1 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:56.097353 11662 raft_consensus.cc:2932] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.101393 14540 log.cc:824] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:56.101928 11120 catalog_manager.cc:5484] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.113718 14560 tablet_copy_client.cc:523] T 93602521de174997bda6702d25aafce1 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:56.116508 14560 tablet_bootstrap.cc:492] T 93602521de174997bda6702d25aafce1 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap starting.
I20240501 07:59:56.125059 14560 log.cc:824] T 93602521de174997bda6702d25aafce1 P 53d1d2c0767b4f6195b69758a3dd6cf9: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:56.132460 11401 consensus_queue.cc:237] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:56.134356 11662 raft_consensus.cc:1261] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.134858 14068 consensus_queue.cc:1035] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.137073 11531 raft_consensus.cc:1261] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.137753 14457 consensus_queue.cc:1035] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:56.138576 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f6e4ae97e884219ac7d0d94997d9cc9. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:56.141988 14540 tablet_bootstrap.cc:492] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:56.142463 14540 tablet_bootstrap.cc:492] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608: Bootstrap complete.
I20240501 07:59:56.142834 14540 ts_tablet_manager.cc:1367] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608: Time spent bootstrapping tablet: real 0.061s user 0.031s sys 0.009s
I20240501 07:59:56.145552 14540 raft_consensus.cc:348] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:56.145889 14540 raft_consensus.cc:729] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca177138f2c44d2eafa1646b795f7608, State: Initialized, Role: FOLLOWER
I20240501 07:59:56.146052 14457 raft_consensus.cc:2932] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.146579 14540 consensus_queue.cc:260] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:56.147815 14540 ts_tablet_manager.cc:1398] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:59:56.148475 11662 raft_consensus.cc:2932] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.148874 11290 tablet_copy_service.cc:342] P e2dafc1866314669ad0de8fb7bf7fd05: Request end of tablet copy session ca177138f2c44d2eafa1646b795f7608-f2e63d24d7df4fa3b2b26695e70ebe9f received from {username='slave'} at 127.0.99.136:55185
I20240501 07:59:56.148975 11290 tablet_copy_service.cc:433] P e2dafc1866314669ad0de8fb7bf7fd05: ending tablet copy session ca177138f2c44d2eafa1646b795f7608-f2e63d24d7df4fa3b2b26695e70ebe9f on tablet f2e63d24d7df4fa3b2b26695e70ebe9f with peer ca177138f2c44d2eafa1646b795f7608
I20240501 07:59:56.151556 11123 catalog_manager.cc:5484] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.154680 11531 raft_consensus.cc:2932] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.157462 11401 consensus_queue.cc:237] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:56.162817 11401 consensus_queue.cc:237] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:56.164523 11663 raft_consensus.cc:1261] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.165144 14456 consensus_queue.cc:1035] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:56.165812 11337 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90f11b5693d7456d8a604e6c8726e69a. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:56.166988 14560 tablet_bootstrap.cc:492] T 93602521de174997bda6702d25aafce1 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:56.167174 11662 raft_consensus.cc:1261] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.167459 14560 tablet_bootstrap.cc:492] T 93602521de174997bda6702d25aafce1 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap complete.
I20240501 07:59:56.167577 14456 consensus_queue.cc:1035] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:56.167677 11336 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e8550ea95984d08937b9b3e2af12bec. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:56.167752 11530 raft_consensus.cc:1261] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.167713 11661 consensus_queue.cc:237] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:56.168249 14560 ts_tablet_manager.cc:1367] T 93602521de174997bda6702d25aafce1 P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent bootstrapping tablet: real 0.052s user 0.039s sys 0.000s
I20240501 07:59:56.168370 14456 consensus_queue.cc:1035] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.169349 11531 raft_consensus.cc:1261] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.169721 14456 consensus_queue.cc:1035] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.169783 11270 raft_consensus.cc:1261] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.170328 14641 consensus_queue.cc:1035] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.170679 11532 raft_consensus.cc:1261] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.171139 13507 consensus_queue.cc:1035] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.172019 14560 raft_consensus.cc:348] T 93602521de174997bda6702d25aafce1 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:56.172303 14560 raft_consensus.cc:729] T 93602521de174997bda6702d25aafce1 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d1d2c0767b4f6195b69758a3dd6cf9, State: Initialized, Role: FOLLOWER
I20240501 07:59:56.173039 14560 consensus_queue.cc:260] T 93602521de174997bda6702d25aafce1 P 53d1d2c0767b4f6195b69758a3dd6cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:56.174001 14560 ts_tablet_manager.cc:1398] T 93602521de174997bda6702d25aafce1 P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 07:59:56.174976 11552 tablet_copy_service.cc:342] P 22ec12774a4047a8b859743dc73f8ff6: Request end of tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-93602521de174997bda6702d25aafce1 received from {username='slave'} at 127.0.99.134:40967
I20240501 07:59:56.175065 11552 tablet_copy_service.cc:433] P 22ec12774a4047a8b859743dc73f8ff6: ending tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-93602521de174997bda6702d25aafce1 on tablet 93602521de174997bda6702d25aafce1 with peer 53d1d2c0767b4f6195b69758a3dd6cf9
I20240501 07:59:56.175869 11532 raft_consensus.cc:2932] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:56.176092 14641 raft_consensus.cc:2932] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:56.176887 14456 raft_consensus.cc:2932] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:56.179749 11662 raft_consensus.cc:2932] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:56.181751 11532 raft_consensus.cc:2932] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:56.183548 11270 raft_consensus.cc:2932] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:56.184216 11123 catalog_manager.cc:5484] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:56.184216 11120 catalog_manager.cc:5484] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
W20240501 07:59:56.191632 11596 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771): Couldn't send request to peer 53d1d2c0767b4f6195b69758a3dd6cf9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f80db2e37bbc4716ad3a5200c61ee97b. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:56.194577 14659 ts_tablet_manager.cc:902] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336: Initiating tablet copy from peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 07:59:56.194487 11401 consensus_queue.cc:237] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:56.194203 14430 raft_consensus.cc:2932] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.195003 11530 raft_consensus.cc:2932] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.195502 14659 tablet_copy_client.cc:308] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46085
I20240501 07:59:56.197597 11269 raft_consensus.cc:1261] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.198607 14456 consensus_queue.cc:1035] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.198943 11530 raft_consensus.cc:1261] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:56.199371 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 164fd265561e47128e7ff585b8cfab07. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:56.199797 11125 catalog_manager.cc:5484] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.200897 14456 consensus_queue.cc:1035] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.203119 11662 raft_consensus.cc:2932] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.206748 11290 tablet_copy_service.cc:140] P e2dafc1866314669ad0de8fb7bf7fd05: Received BeginTabletCopySession request for tablet bd8348990f1b496bb62d8c0484193442 from peer e5c03261d31c4d9a91077a17e1cd0336 ({username='slave'} at 127.0.99.135:33281)
I20240501 07:59:56.206876 11290 tablet_copy_service.cc:161] P e2dafc1866314669ad0de8fb7bf7fd05: Beginning new tablet copy session on tablet bd8348990f1b496bb62d8c0484193442 from peer e5c03261d31c4d9a91077a17e1cd0336 at {username='slave'} at 127.0.99.135:33281: session id = e5c03261d31c4d9a91077a17e1cd0336-bd8348990f1b496bb62d8c0484193442
I20240501 07:59:56.207782 11290 tablet_copy_source_session.cc:205] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:56.213409 14659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8348990f1b496bb62d8c0484193442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:56.213781 14678 raft_consensus.cc:2932] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.214933 11530 raft_consensus.cc:2932] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.216871 11269 raft_consensus.cc:2932] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.222991 11120 catalog_manager.cc:5484] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.230412 14659 tablet_copy_client.cc:783] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:56.230811 14659 tablet_copy_client.cc:647] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:56.232578 14659 tablet_copy_client.cc:523] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:56.234869 14659 tablet_bootstrap.cc:492] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap starting.
I20240501 07:59:56.242238 14659 log.cc:824] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:56.243126 11401 consensus_queue.cc:237] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:56.243316 11269 consensus_queue.cc:237] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:56.245296 11530 raft_consensus.cc:1261] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:56.245901 11206 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 867ba08a15b44d4c9e6fa92b0d518d1d. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:56.246136 11662 raft_consensus.cc:1261] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.246454 14289 consensus_queue.cc:1035] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.246621 14241 consensus_queue.cc:1035] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.248203 11269 raft_consensus.cc:1261] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:56.248275 11337 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9431cf5853c4148ac38855b1c083d4f. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:56.248658 14430 consensus_queue.cc:1035] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.252354 11532 raft_consensus.cc:1261] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.252895 14430 consensus_queue.cc:1035] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.255270 14289 raft_consensus.cc:2932] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:56.256067 14429 raft_consensus.cc:2932] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:56.256572 11532 raft_consensus.cc:2932] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:56.258927 11125 catalog_manager.cc:5484] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:56.259472 11662 raft_consensus.cc:2932] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:56.262786 11123 catalog_manager.cc:5484] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:56.265013 11532 raft_consensus.cc:2932] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:56.266144 11269 raft_consensus.cc:2932] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:56.276278 14560 ts_tablet_manager.cc:902] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9: Initiating tablet copy from peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 07:59:56.276762 14560 tablet_copy_client.cc:308] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46085
I20240501 07:59:56.284294 11290 tablet_copy_service.cc:140] P e2dafc1866314669ad0de8fb7bf7fd05: Received BeginTabletCopySession request for tablet 25cee7e51c7c438aa721d71fb82dc71e from peer 53d1d2c0767b4f6195b69758a3dd6cf9 ({username='slave'} at 127.0.99.134:37773)
I20240501 07:59:56.284426 11290 tablet_copy_service.cc:161] P e2dafc1866314669ad0de8fb7bf7fd05: Beginning new tablet copy session on tablet 25cee7e51c7c438aa721d71fb82dc71e from peer 53d1d2c0767b4f6195b69758a3dd6cf9 at {username='slave'} at 127.0.99.134:37773: session id = 53d1d2c0767b4f6195b69758a3dd6cf9-25cee7e51c7c438aa721d71fb82dc71e
I20240501 07:59:56.285427 11290 tablet_copy_source_session.cc:205] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:56.286509 14560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25cee7e51c7c438aa721d71fb82dc71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:56.287144 14659 tablet_bootstrap.cc:492] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:56.287501 14659 tablet_bootstrap.cc:492] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap complete.
I20240501 07:59:56.287818 14659 ts_tablet_manager.cc:1367] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336: Time spent bootstrapping tablet: real 0.053s user 0.032s sys 0.003s
I20240501 07:59:56.288895 14560 tablet_copy_client.cc:783] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:56.289047 14560 tablet_copy_client.cc:647] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:56.290366 14659 raft_consensus.cc:348] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:56.290776 14659 raft_consensus.cc:729] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c03261d31c4d9a91077a17e1cd0336, State: Initialized, Role: FOLLOWER
I20240501 07:59:56.291355 14659 consensus_queue.cc:260] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:56.292696 14659 ts_tablet_manager.cc:1398] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 07:59:56.294517 11290 tablet_copy_service.cc:342] P e2dafc1866314669ad0de8fb7bf7fd05: Request end of tablet copy session e5c03261d31c4d9a91077a17e1cd0336-bd8348990f1b496bb62d8c0484193442 received from {username='slave'} at 127.0.99.135:33281
I20240501 07:59:56.294611 11290 tablet_copy_service.cc:433] P e2dafc1866314669ad0de8fb7bf7fd05: ending tablet copy session e5c03261d31c4d9a91077a17e1cd0336-bd8348990f1b496bb62d8c0484193442 on tablet bd8348990f1b496bb62d8c0484193442 with peer e5c03261d31c4d9a91077a17e1cd0336
I20240501 07:59:56.297186 14560 tablet_copy_client.cc:523] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:56.298825 14560 tablet_bootstrap.cc:492] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap starting.
I20240501 07:59:56.337352 14560 tablet_bootstrap.cc:492] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:56.343956 14560 tablet_bootstrap.cc:492] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap complete.
I20240501 07:59:56.344223 14560 ts_tablet_manager.cc:1367] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent bootstrapping tablet: real 0.045s user 0.025s sys 0.001s
I20240501 07:59:56.344939 14560 raft_consensus.cc:348] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:56.345080 14560 raft_consensus.cc:729] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d1d2c0767b4f6195b69758a3dd6cf9, State: Initialized, Role: FOLLOWER
I20240501 07:59:56.345253 14560 consensus_queue.cc:260] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:56.346693 14560 ts_tablet_manager.cc:1398] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:56.347554 11290 tablet_copy_service.cc:342] P e2dafc1866314669ad0de8fb7bf7fd05: Request end of tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-25cee7e51c7c438aa721d71fb82dc71e received from {username='slave'} at 127.0.99.134:37773
I20240501 07:59:56.347637 11290 tablet_copy_service.cc:433] P e2dafc1866314669ad0de8fb7bf7fd05: ending tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-25cee7e51c7c438aa721d71fb82dc71e on tablet 25cee7e51c7c438aa721d71fb82dc71e with peer 53d1d2c0767b4f6195b69758a3dd6cf9
I20240501 07:59:56.351745 11662 consensus_queue.cc:237] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:56.355016 11401 raft_consensus.cc:1261] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.356606 11532 raft_consensus.cc:1261] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.357259 14640 consensus_queue.cc:1035] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.357966 14640 consensus_queue.cc:1035] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:56.361713 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f61e70ba02384f81b9ae36b1c65465b9. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:56.365232 14640 raft_consensus.cc:2932] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:56.368245 11532 raft_consensus.cc:2932] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:56.369434 11662 consensus_queue.cc:237] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:56.370344 11123 catalog_manager.cc:5484] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 reported cstate change: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:56.371510 11269 raft_consensus.cc:1261] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.371619 11532 raft_consensus.cc:1261] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.371884 11401 raft_consensus.cc:2932] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:56.376473 14660 consensus_queue.cc:1035] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.377786 14660 consensus_queue.cc:1035] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:56.378876 11596 consensus_peers.cc:485] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 -> Peer 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771): Couldn't send request to peer 53d1d2c0767b4f6195b69758a3dd6cf9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b7de08d2f004cd8abc2bbe1cf7b1444. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:56.387909 13507 raft_consensus.cc:2932] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:56.388458 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:56.391290 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:56.391350 11532 raft_consensus.cc:2932] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:56.391496 11120 catalog_manager.cc:5484] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 reported cstate change: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:56.395331 11269 raft_consensus.cc:2932] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:56.397284 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:56.400132 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:56.413568 14659 ts_tablet_manager.cc:902] T 18b146cc4e164e1d9f9f827217c9b3ab P e5c03261d31c4d9a91077a17e1cd0336: Initiating tablet copy from peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 07:59:56.414069 14659 tablet_copy_client.cc:308] T 18b146cc4e164e1d9f9f827217c9b3ab P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46085
I20240501 07:59:56.414695 11290 tablet_copy_service.cc:140] P e2dafc1866314669ad0de8fb7bf7fd05: Received BeginTabletCopySession request for tablet 18b146cc4e164e1d9f9f827217c9b3ab from peer e5c03261d31c4d9a91077a17e1cd0336 ({username='slave'} at 127.0.99.135:33281)
I20240501 07:59:56.414810 11290 tablet_copy_service.cc:161] P e2dafc1866314669ad0de8fb7bf7fd05: Beginning new tablet copy session on tablet 18b146cc4e164e1d9f9f827217c9b3ab from peer e5c03261d31c4d9a91077a17e1cd0336 at {username='slave'} at 127.0.99.135:33281: session id = e5c03261d31c4d9a91077a17e1cd0336-18b146cc4e164e1d9f9f827217c9b3ab
I20240501 07:59:56.415768 11290 tablet_copy_source_session.cc:205] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:56.418179 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:56.420431 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:56.421571 14659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b146cc4e164e1d9f9f827217c9b3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:56.422452 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:56.427642 14659 tablet_copy_client.cc:783] T 18b146cc4e164e1d9f9f827217c9b3ab P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:56.428001 14659 tablet_copy_client.cc:647] T 18b146cc4e164e1d9f9f827217c9b3ab P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:56.434919 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:56.454041 14659 tablet_copy_client.cc:523] T 18b146cc4e164e1d9f9f827217c9b3ab P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 07:59:56.455503 14839 log.cc:924] Time spent T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20240501 07:59:56.456936 14659 tablet_bootstrap.cc:492] T 18b146cc4e164e1d9f9f827217c9b3ab P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap starting.
I20240501 07:59:56.490341 14893 ts_tablet_manager.cc:902] T 64613ea537c842f29d810e829278394e P 60e8857abe7c41038af46ebb5a095a90: Initiating tablet copy from peer 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:56.491349 14893 tablet_copy_client.cc:308] T 64613ea537c842f29d810e829278394e P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41575
I20240501 07:59:56.501271 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:56.512920 11421 tablet_copy_service.cc:140] P 655ebda1fb2c49c4af7087a760529835: Received BeginTabletCopySession request for tablet 64613ea537c842f29d810e829278394e from peer 60e8857abe7c41038af46ebb5a095a90 ({username='slave'} at 127.0.99.133:38419)
I20240501 07:59:56.513123 11421 tablet_copy_service.cc:161] P 655ebda1fb2c49c4af7087a760529835: Beginning new tablet copy session on tablet 64613ea537c842f29d810e829278394e from peer 60e8857abe7c41038af46ebb5a095a90 at {username='slave'} at 127.0.99.133:38419: session id = 60e8857abe7c41038af46ebb5a095a90-64613ea537c842f29d810e829278394e
I20240501 07:59:56.515235 11421 tablet_copy_source_session.cc:205] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:56.519907 14893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64613ea537c842f29d810e829278394e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:56.528163 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:56.533665 13449 raft_consensus.cc:1203] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:56.544201 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:56.547276 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:56.555771 14893 tablet_copy_client.cc:783] T 64613ea537c842f29d810e829278394e P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:56.556471 14893 tablet_copy_client.cc:647] T 64613ea537c842f29d810e829278394e P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:56.558779 14893 tablet_copy_client.cc:523] T 64613ea537c842f29d810e829278394e P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:56.562233 14893 tablet_bootstrap.cc:492] T 64613ea537c842f29d810e829278394e P 60e8857abe7c41038af46ebb5a095a90: Bootstrap starting.
I20240501 07:59:56.563243 14560 ts_tablet_manager.cc:902] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9: Initiating tablet copy from peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:56.563738 14560 tablet_copy_client.cc:308] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:44403
I20240501 07:59:56.568161 14918 ts_tablet_manager.cc:902] T 6d214be86534458091f54a386d7d3334 P 60e8857abe7c41038af46ebb5a095a90: Initiating tablet copy from peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:56.568792 14918 tablet_copy_client.cc:308] T 6d214be86534458091f54a386d7d3334 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37957
I20240501 07:59:56.572134 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:56.572883 11683 tablet_copy_service.cc:140] P 603919bb02364c04b6c530245b8b02f2: Received BeginTabletCopySession request for tablet f80db2e37bbc4716ad3a5200c61ee97b from peer 53d1d2c0767b4f6195b69758a3dd6cf9 ({username='slave'} at 127.0.99.134:57705)
I20240501 07:59:56.573045 11683 tablet_copy_service.cc:161] P 603919bb02364c04b6c530245b8b02f2: Beginning new tablet copy session on tablet f80db2e37bbc4716ad3a5200c61ee97b from peer 53d1d2c0767b4f6195b69758a3dd6cf9 at {username='slave'} at 127.0.99.134:57705: session id = 53d1d2c0767b4f6195b69758a3dd6cf9-f80db2e37bbc4716ad3a5200c61ee97b
I20240501 07:59:56.574539 11683 tablet_copy_source_session.cc:205] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:56.575820 14560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80db2e37bbc4716ad3a5200c61ee97b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:56.578145 14560 tablet_copy_client.cc:783] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:56.578305 14560 tablet_copy_client.cc:647] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:56.578815 11552 tablet_copy_service.cc:140] P 22ec12774a4047a8b859743dc73f8ff6: Received BeginTabletCopySession request for tablet 6d214be86534458091f54a386d7d3334 from peer 60e8857abe7c41038af46ebb5a095a90 ({username='slave'} at 127.0.99.133:59721)
I20240501 07:59:56.578917 11552 tablet_copy_service.cc:161] P 22ec12774a4047a8b859743dc73f8ff6: Beginning new tablet copy session on tablet 6d214be86534458091f54a386d7d3334 from peer 60e8857abe7c41038af46ebb5a095a90 at {username='slave'} at 127.0.99.133:59721: session id = 60e8857abe7c41038af46ebb5a095a90-6d214be86534458091f54a386d7d3334
I20240501 07:59:56.579825 11552 tablet_copy_source_session.cc:205] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:56.580221 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:56.580839 11663 consensus_queue.cc:237] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:56.582599 11269 raft_consensus.cc:1261] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.583039 14641 consensus_queue.cc:1035] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.583611 11532 raft_consensus.cc:1261] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.584260 14918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d214be86534458091f54a386d7d3334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:56.584271 14641 consensus_queue.cc:1035] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.586736 14918 tablet_copy_client.cc:783] T 6d214be86534458091f54a386d7d3334 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:56.586949 14918 tablet_copy_client.cc:647] T 6d214be86534458091f54a386d7d3334 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:56.589983 14659 tablet_bootstrap.cc:492] T 18b146cc4e164e1d9f9f827217c9b3ab P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:56.590262 14659 tablet_bootstrap.cc:492] T 18b146cc4e164e1d9f9f827217c9b3ab P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap complete.
W20240501 07:59:56.590397 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed88ea311f474ab28cc6fbe6069258bf. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:56.591038 14659 ts_tablet_manager.cc:1367] T 18b146cc4e164e1d9f9f827217c9b3ab P e5c03261d31c4d9a91077a17e1cd0336: Time spent bootstrapping tablet: real 0.134s user 0.030s sys 0.001s
I20240501 07:59:56.592666 14659 raft_consensus.cc:348] T 18b146cc4e164e1d9f9f827217c9b3ab P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:56.592800 14659 raft_consensus.cc:729] T 18b146cc4e164e1d9f9f827217c9b3ab P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c03261d31c4d9a91077a17e1cd0336, State: Initialized, Role: FOLLOWER
I20240501 07:59:56.592962 14659 consensus_queue.cc:260] T 18b146cc4e164e1d9f9f827217c9b3ab P e5c03261d31c4d9a91077a17e1cd0336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:56.593690 14659 ts_tablet_manager.cc:1398] T 18b146cc4e164e1d9f9f827217c9b3ab P e5c03261d31c4d9a91077a17e1cd0336: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:56.596500 11290 tablet_copy_service.cc:342] P e2dafc1866314669ad0de8fb7bf7fd05: Request end of tablet copy session e5c03261d31c4d9a91077a17e1cd0336-18b146cc4e164e1d9f9f827217c9b3ab received from {username='slave'} at 127.0.99.135:33281
I20240501 07:59:56.596591 11290 tablet_copy_service.cc:433] P e2dafc1866314669ad0de8fb7bf7fd05: ending tablet copy session e5c03261d31c4d9a91077a17e1cd0336-18b146cc4e164e1d9f9f827217c9b3ab on tablet 18b146cc4e164e1d9f9f827217c9b3ab with peer e5c03261d31c4d9a91077a17e1cd0336
I20240501 07:59:56.601940 14641 raft_consensus.cc:2932] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.602532 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:56.605160 11269 raft_consensus.cc:2932] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.606665 14893 log.cc:824] T 64613ea537c842f29d810e829278394e P 60e8857abe7c41038af46ebb5a095a90: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:56.608321 11532 raft_consensus.cc:2932] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.611042 13186 raft_consensus.cc:1203] T 93602521de174997bda6702d25aafce1 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:56.611760 11121 catalog_manager.cc:5484] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.614675 14560 tablet_copy_client.cc:523] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:56.617250 14560 tablet_bootstrap.cc:492] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap starting.
I20240501 07:59:56.624274 14918 tablet_copy_client.cc:523] T 6d214be86534458091f54a386d7d3334 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:56.625876 14918 tablet_bootstrap.cc:492] T 6d214be86534458091f54a386d7d3334 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap starting.
I20240501 07:59:56.636109 14540 ts_tablet_manager.cc:902] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608: Initiating tablet copy from peer 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:56.637559 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:56.638082 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:56.642454 14540 tablet_copy_client.cc:308] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41575
I20240501 07:59:56.643615 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:56.657714 14659 ts_tablet_manager.cc:902] T 64613ea537c842f29d810e829278394e P e5c03261d31c4d9a91077a17e1cd0336: Initiating tablet copy from peer 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:56.658195 14659 tablet_copy_client.cc:308] T 64613ea537c842f29d810e829278394e P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41575
I20240501 07:59:56.662557 14958 ts_tablet_manager.cc:902] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 60e8857abe7c41038af46ebb5a095a90: Initiating tablet copy from peer 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:56.663259 14958 tablet_copy_client.cc:308] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41575
I20240501 07:59:56.663727 11421 tablet_copy_service.cc:140] P 655ebda1fb2c49c4af7087a760529835: Received BeginTabletCopySession request for tablet 64613ea537c842f29d810e829278394e from peer e5c03261d31c4d9a91077a17e1cd0336 ({username='slave'} at 127.0.99.135:48791)
I20240501 07:59:56.663779 11420 tablet_copy_service.cc:140] P 655ebda1fb2c49c4af7087a760529835: Received BeginTabletCopySession request for tablet 3f6e4ae97e884219ac7d0d94997d9cc9 from peer 60e8857abe7c41038af46ebb5a095a90 ({username='slave'} at 127.0.99.133:38419)
I20240501 07:59:56.663841 11421 tablet_copy_service.cc:161] P 655ebda1fb2c49c4af7087a760529835: Beginning new tablet copy session on tablet 64613ea537c842f29d810e829278394e from peer e5c03261d31c4d9a91077a17e1cd0336 at {username='slave'} at 127.0.99.135:48791: session id = e5c03261d31c4d9a91077a17e1cd0336-64613ea537c842f29d810e829278394e
I20240501 07:59:56.664894 11421 tablet_copy_source_session.cc:205] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:56.665206 11420 tablet_copy_service.cc:161] P 655ebda1fb2c49c4af7087a760529835: Beginning new tablet copy session on tablet 3f6e4ae97e884219ac7d0d94997d9cc9 from peer 60e8857abe7c41038af46ebb5a095a90 at {username='slave'} at 127.0.99.133:38419: session id = 60e8857abe7c41038af46ebb5a095a90-3f6e4ae97e884219ac7d0d94997d9cc9
I20240501 07:59:56.666148 11420 tablet_copy_source_session.cc:205] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:56.667588 14659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64613ea537c842f29d810e829278394e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:56.669802 14659 tablet_copy_client.cc:783] T 64613ea537c842f29d810e829278394e P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:56.669962 14659 tablet_copy_client.cc:647] T 64613ea537c842f29d810e829278394e P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:56.675949 14988 ts_tablet_manager.cc:902] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608: Initiating tablet copy from peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 07:59:56.677290 11269 consensus_queue.cc:237] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:56.679100 14918 tablet_bootstrap.cc:492] T 6d214be86534458091f54a386d7d3334 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:56.679260 14958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6e4ae97e884219ac7d0d94997d9cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:56.679474 14918 tablet_bootstrap.cc:492] T 6d214be86534458091f54a386d7d3334 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap complete.
I20240501 07:59:56.679797 14918 ts_tablet_manager.cc:1367] T 6d214be86534458091f54a386d7d3334 P 60e8857abe7c41038af46ebb5a095a90: Time spent bootstrapping tablet: real 0.054s user 0.032s sys 0.000s
I20240501 07:59:56.681126 11401 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:56.682194 14958 tablet_copy_client.cc:783] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:56.682387 11269 consensus_queue.cc:237] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:56.682386 14918 raft_consensus.cc:348] T 6d214be86534458091f54a386d7d3334 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:56.683017 14918 raft_consensus.cc:729] T 6d214be86534458091f54a386d7d3334 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e8857abe7c41038af46ebb5a095a90, State: Initialized, Role: FOLLOWER
I20240501 07:59:56.683542 14918 consensus_queue.cc:260] T 6d214be86534458091f54a386d7d3334 P 60e8857abe7c41038af46ebb5a095a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:56.684195 11532 raft_consensus.cc:1261] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.684547 14918 ts_tablet_manager.cc:1398] T 6d214be86534458091f54a386d7d3334 P 60e8857abe7c41038af46ebb5a095a90: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 07:59:56.684695 13728 consensus_queue.cc:1035] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.685254 11552 tablet_copy_service.cc:342] P 22ec12774a4047a8b859743dc73f8ff6: Request end of tablet copy session 60e8857abe7c41038af46ebb5a095a90-6d214be86534458091f54a386d7d3334 received from {username='slave'} at 127.0.99.133:59721
I20240501 07:59:56.685331 11552 tablet_copy_service.cc:433] P 22ec12774a4047a8b859743dc73f8ff6: ending tablet copy session 60e8857abe7c41038af46ebb5a095a90-6d214be86534458091f54a386d7d3334 on tablet 6d214be86534458091f54a386d7d3334 with peer 60e8857abe7c41038af46ebb5a095a90
I20240501 07:59:56.686530 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:56.686966 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:56.687057 11401 raft_consensus.cc:1261] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.687525 13728 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:56.689146 13728 consensus_queue.cc:1035] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.682538 14958 tablet_copy_client.cc:647] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:56.692288 13186 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:56.692941 14659 tablet_copy_client.cc:523] T 64613ea537c842f29d810e829278394e P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 07:59:56.693769 11203 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771): Couldn't send request to peer 53d1d2c0767b4f6195b69758a3dd6cf9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea97f34590684bcf998228245dbcd9b0. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:56.694000 14290 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:56.694766 14659 tablet_bootstrap.cc:492] T 64613ea537c842f29d810e829278394e P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap starting.
I20240501 07:59:56.696380 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:56.698784 11663 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:56.699592 14290 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:56.702145 14290 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.707051 11401 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.707368 11125 catalog_manager.cc:5484] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index 4 to 5, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.709578 11663 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.712826 14918 ts_tablet_manager.cc:902] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90: Initiating tablet copy from peer 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:56.713598 14958 tablet_copy_client.cc:523] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:56.715512 14958 tablet_bootstrap.cc:492] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap starting.
I20240501 07:59:56.718503 14988 tablet_copy_client.cc:308] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46085
I20240501 07:59:56.719063 11290 tablet_copy_service.cc:140] P e2dafc1866314669ad0de8fb7bf7fd05: Received BeginTabletCopySession request for tablet 867ba08a15b44d4c9e6fa92b0d518d1d from peer ca177138f2c44d2eafa1646b795f7608 ({username='slave'} at 127.0.99.136:55185)
I20240501 07:59:56.719219 11290 tablet_copy_service.cc:161] P e2dafc1866314669ad0de8fb7bf7fd05: Beginning new tablet copy session on tablet 867ba08a15b44d4c9e6fa92b0d518d1d from peer ca177138f2c44d2eafa1646b795f7608 at {username='slave'} at 127.0.99.136:55185: session id = ca177138f2c44d2eafa1646b795f7608-867ba08a15b44d4c9e6fa92b0d518d1d
I20240501 07:59:56.719841 14918 tablet_copy_client.cc:308] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41575
I20240501 07:59:56.720144 11290 tablet_copy_source_session.cc:205] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:56.720427 11420 tablet_copy_service.cc:140] P 655ebda1fb2c49c4af7087a760529835: Received BeginTabletCopySession request for tablet 5e8550ea95984d08937b9b3e2af12bec from peer 60e8857abe7c41038af46ebb5a095a90 ({username='slave'} at 127.0.99.133:38419)
I20240501 07:59:56.720538 11420 tablet_copy_service.cc:161] P 655ebda1fb2c49c4af7087a760529835: Beginning new tablet copy session on tablet 5e8550ea95984d08937b9b3e2af12bec from peer 60e8857abe7c41038af46ebb5a095a90 at {username='slave'} at 127.0.99.133:38419: session id = 60e8857abe7c41038af46ebb5a095a90-5e8550ea95984d08937b9b3e2af12bec
I20240501 07:59:56.721127 14988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 867ba08a15b44d4c9e6fa92b0d518d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:56.721419 11420 tablet_copy_source_session.cc:205] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:56.722504 14918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8550ea95984d08937b9b3e2af12bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:56.723716 14988 tablet_copy_client.cc:783] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:56.723863 14988 tablet_copy_client.cc:647] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:56.724802 14918 tablet_copy_client.cc:783] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:56.725028 14918 tablet_copy_client.cc:647] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:56.730002 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:56.735821 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:56.737542 11420 tablet_copy_service.cc:140] P 655ebda1fb2c49c4af7087a760529835: Received BeginTabletCopySession request for tablet 90f11b5693d7456d8a604e6c8726e69a from peer ca177138f2c44d2eafa1646b795f7608 ({username='slave'} at 127.0.99.136:50157)
I20240501 07:59:56.737673 11420 tablet_copy_service.cc:161] P 655ebda1fb2c49c4af7087a760529835: Beginning new tablet copy session on tablet 90f11b5693d7456d8a604e6c8726e69a from peer ca177138f2c44d2eafa1646b795f7608 at {username='slave'} at 127.0.99.136:50157: session id = ca177138f2c44d2eafa1646b795f7608-90f11b5693d7456d8a604e6c8726e69a
I20240501 07:59:56.738595 11420 tablet_copy_source_session.cc:205] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:56.739408 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:56.739460 14540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f11b5693d7456d8a604e6c8726e69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:56.741739 14540 tablet_copy_client.cc:783] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:56.741896 14540 tablet_copy_client.cc:647] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:56.743791 14979 ts_tablet_manager.cc:902] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608: Initiating tablet copy from peer 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:56.744355 14979 tablet_copy_client.cc:308] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41575
I20240501 07:59:56.744750 11420 tablet_copy_service.cc:140] P 655ebda1fb2c49c4af7087a760529835: Received BeginTabletCopySession request for tablet b9431cf5853c4148ac38855b1c083d4f from peer ca177138f2c44d2eafa1646b795f7608 ({username='slave'} at 127.0.99.136:50157)
I20240501 07:59:56.744843 11420 tablet_copy_service.cc:161] P 655ebda1fb2c49c4af7087a760529835: Beginning new tablet copy session on tablet b9431cf5853c4148ac38855b1c083d4f from peer ca177138f2c44d2eafa1646b795f7608 at {username='slave'} at 127.0.99.136:50157: session id = ca177138f2c44d2eafa1646b795f7608-b9431cf5853c4148ac38855b1c083d4f
I20240501 07:59:56.744988 14540 tablet_copy_client.cc:523] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:56.745790 11420 tablet_copy_source_session.cc:205] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:56.746630 14540 tablet_bootstrap.cc:492] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608: Bootstrap starting.
I20240501 07:59:56.746958 14979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9431cf5853c4148ac38855b1c083d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:56.749655 14979 tablet_copy_client.cc:783] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:56.751646 13317 raft_consensus.cc:1203] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:56.756515 14979 tablet_copy_client.cc:647] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:56.763478 14958 tablet_bootstrap.cc:492] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:56.764016 14958 tablet_bootstrap.cc:492] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap complete.
I20240501 07:59:56.764276 14958 ts_tablet_manager.cc:1367] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 60e8857abe7c41038af46ebb5a095a90: Time spent bootstrapping tablet: real 0.049s user 0.026s sys 0.002s
I20240501 07:59:56.765017 14958 raft_consensus.cc:348] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:56.765226 14958 raft_consensus.cc:729] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e8857abe7c41038af46ebb5a095a90, State: Initialized, Role: FOLLOWER
I20240501 07:59:56.765450 14958 consensus_queue.cc:260] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 60e8857abe7c41038af46ebb5a095a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:56.765794 14958 ts_tablet_manager.cc:1398] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 60e8857abe7c41038af46ebb5a095a90: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:56.770642 11420 tablet_copy_service.cc:342] P 655ebda1fb2c49c4af7087a760529835: Request end of tablet copy session 60e8857abe7c41038af46ebb5a095a90-3f6e4ae97e884219ac7d0d94997d9cc9 received from {username='slave'} at 127.0.99.133:38419
I20240501 07:59:56.770785 11420 tablet_copy_service.cc:433] P 655ebda1fb2c49c4af7087a760529835: ending tablet copy session 60e8857abe7c41038af46ebb5a095a90-3f6e4ae97e884219ac7d0d94997d9cc9 on tablet 3f6e4ae97e884219ac7d0d94997d9cc9 with peer 60e8857abe7c41038af46ebb5a095a90
I20240501 07:59:56.772274 14560 tablet_bootstrap.cc:492] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:56.772583 14560 tablet_bootstrap.cc:492] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap complete.
I20240501 07:59:56.772804 14560 ts_tablet_manager.cc:1367] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent bootstrapping tablet: real 0.156s user 0.024s sys 0.000s
I20240501 07:59:56.773313 14560 raft_consensus.cc:348] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:56.773448 14560 raft_consensus.cc:729] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d1d2c0767b4f6195b69758a3dd6cf9, State: Initialized, Role: FOLLOWER
I20240501 07:59:56.773623 14560 consensus_queue.cc:260] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:56.774027 14560 ts_tablet_manager.cc:1398] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 07:59:56.774740 11683 tablet_copy_service.cc:342] P 603919bb02364c04b6c530245b8b02f2: Request end of tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-f80db2e37bbc4716ad3a5200c61ee97b received from {username='slave'} at 127.0.99.134:57705
I20240501 07:59:56.774853 11683 tablet_copy_service.cc:433] P 603919bb02364c04b6c530245b8b02f2: ending tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-f80db2e37bbc4716ad3a5200c61ee97b on tablet f80db2e37bbc4716ad3a5200c61ee97b with peer 53d1d2c0767b4f6195b69758a3dd6cf9
I20240501 07:59:56.778092 14958 ts_tablet_manager.cc:902] T 164fd265561e47128e7ff585b8cfab07 P 60e8857abe7c41038af46ebb5a095a90: Initiating tablet copy from peer 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:56.778610 14958 tablet_copy_client.cc:308] T 164fd265561e47128e7ff585b8cfab07 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41575
I20240501 07:59:56.779187 11420 tablet_copy_service.cc:140] P 655ebda1fb2c49c4af7087a760529835: Received BeginTabletCopySession request for tablet 164fd265561e47128e7ff585b8cfab07 from peer 60e8857abe7c41038af46ebb5a095a90 ({username='slave'} at 127.0.99.133:38419)
I20240501 07:59:56.779318 11420 tablet_copy_service.cc:161] P 655ebda1fb2c49c4af7087a760529835: Beginning new tablet copy session on tablet 164fd265561e47128e7ff585b8cfab07 from peer 60e8857abe7c41038af46ebb5a095a90 at {username='slave'} at 127.0.99.133:38419: session id = 60e8857abe7c41038af46ebb5a095a90-164fd265561e47128e7ff585b8cfab07
I20240501 07:59:56.780251 11420 tablet_copy_source_session.cc:205] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:56.780234 14659 tablet_bootstrap.cc:492] T 64613ea537c842f29d810e829278394e P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:56.780560 14659 tablet_bootstrap.cc:492] T 64613ea537c842f29d810e829278394e P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap complete.
I20240501 07:59:56.780761 14659 ts_tablet_manager.cc:1367] T 64613ea537c842f29d810e829278394e P e5c03261d31c4d9a91077a17e1cd0336: Time spent bootstrapping tablet: real 0.086s user 0.026s sys 0.000s
I20240501 07:59:56.781208 14958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164fd265561e47128e7ff585b8cfab07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:56.781298 14659 raft_consensus.cc:348] T 64613ea537c842f29d810e829278394e P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:56.781425 14659 raft_consensus.cc:729] T 64613ea537c842f29d810e829278394e P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c03261d31c4d9a91077a17e1cd0336, State: Initialized, Role: FOLLOWER
I20240501 07:59:56.781571 14659 consensus_queue.cc:260] T 64613ea537c842f29d810e829278394e P e5c03261d31c4d9a91077a17e1cd0336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:56.782248 13728 raft_consensus.cc:2932] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:56.784808 11532 raft_consensus.cc:2932] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:56.785050 13185 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:56.788457 11401 raft_consensus.cc:2932] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:56.788537 11125 catalog_manager.cc:5484] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:56.792495 14918 tablet_copy_client.cc:523] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:56.794109 14918 tablet_bootstrap.cc:492] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90: Bootstrap starting.
W20240501 07:59:56.806789 11205 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25cee7e51c7c438aa721d71fb82dc71e. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:56.807514 14958 tablet_copy_client.cc:783] T 164fd265561e47128e7ff585b8cfab07 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:56.807734 14958 tablet_copy_client.cc:647] T 164fd265561e47128e7ff585b8cfab07 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:56.808903 14958 tablet_copy_client.cc:523] T 164fd265561e47128e7ff585b8cfab07 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:56.808832 14659 ts_tablet_manager.cc:1398] T 64613ea537c842f29d810e829278394e P e5c03261d31c4d9a91077a17e1cd0336: Time spent starting tablet: real 0.028s user 0.004s sys 0.000s
I20240501 07:59:56.811163 11420 tablet_copy_service.cc:342] P 655ebda1fb2c49c4af7087a760529835: Request end of tablet copy session e5c03261d31c4d9a91077a17e1cd0336-64613ea537c842f29d810e829278394e received from {username='slave'} at 127.0.99.135:48791
I20240501 07:59:56.811256 11420 tablet_copy_service.cc:433] P 655ebda1fb2c49c4af7087a760529835: ending tablet copy session e5c03261d31c4d9a91077a17e1cd0336-64613ea537c842f29d810e829278394e on tablet 64613ea537c842f29d810e829278394e with peer e5c03261d31c4d9a91077a17e1cd0336
I20240501 07:59:56.813169 14659 ts_tablet_manager.cc:902] T f61e70ba02384f81b9ae36b1c65465b9 P e5c03261d31c4d9a91077a17e1cd0336: Initiating tablet copy from peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:56.813572 14659 tablet_copy_client.cc:308] T f61e70ba02384f81b9ae36b1c65465b9 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:44403
I20240501 07:59:56.812530 14958 tablet_bootstrap.cc:492] T 164fd265561e47128e7ff585b8cfab07 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap starting.
I20240501 07:59:56.820012 14979 tablet_copy_client.cc:523] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:56.820180 11683 tablet_copy_service.cc:140] P 603919bb02364c04b6c530245b8b02f2: Received BeginTabletCopySession request for tablet f61e70ba02384f81b9ae36b1c65465b9 from peer e5c03261d31c4d9a91077a17e1cd0336 ({username='slave'} at 127.0.99.135:57735)
I20240501 07:59:56.820302 11683 tablet_copy_service.cc:161] P 603919bb02364c04b6c530245b8b02f2: Beginning new tablet copy session on tablet f61e70ba02384f81b9ae36b1c65465b9 from peer e5c03261d31c4d9a91077a17e1cd0336 at {username='slave'} at 127.0.99.135:57735: session id = e5c03261d31c4d9a91077a17e1cd0336-f61e70ba02384f81b9ae36b1c65465b9
I20240501 07:59:56.821199 11683 tablet_copy_source_session.cc:205] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:56.821477 14979 tablet_bootstrap.cc:492] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608: Bootstrap starting.
I20240501 07:59:56.823063 14988 tablet_copy_client.cc:523] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:56.823376 14659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61e70ba02384f81b9ae36b1c65465b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:56.824445 14988 tablet_bootstrap.cc:492] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608: Bootstrap starting.
I20240501 07:59:56.826207 14659 tablet_copy_client.cc:783] T f61e70ba02384f81b9ae36b1c65465b9 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:56.826438 14659 tablet_copy_client.cc:647] T f61e70ba02384f81b9ae36b1c65465b9 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:56.836095 14659 tablet_copy_client.cc:523] T f61e70ba02384f81b9ae36b1c65465b9 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:56.837743 14659 tablet_bootstrap.cc:492] T f61e70ba02384f81b9ae36b1c65465b9 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap starting.
I20240501 07:59:56.855194 14979 tablet_bootstrap.cc:492] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:56.855787 14979 tablet_bootstrap.cc:492] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608: Bootstrap complete.
I20240501 07:59:56.856063 14979 ts_tablet_manager.cc:1367] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.005s
I20240501 07:59:56.858522 14979 raft_consensus.cc:348] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:56.858780 14979 raft_consensus.cc:729] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca177138f2c44d2eafa1646b795f7608, State: Initialized, Role: FOLLOWER
I20240501 07:59:56.859068 14979 consensus_queue.cc:260] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:56.860172 14979 ts_tablet_manager.cc:1398] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:56.860759 11420 tablet_copy_service.cc:342] P 655ebda1fb2c49c4af7087a760529835: Request end of tablet copy session ca177138f2c44d2eafa1646b795f7608-b9431cf5853c4148ac38855b1c083d4f received from {username='slave'} at 127.0.99.136:50157
I20240501 07:59:56.860848 11420 tablet_copy_service.cc:433] P 655ebda1fb2c49c4af7087a760529835: ending tablet copy session ca177138f2c44d2eafa1646b795f7608-b9431cf5853c4148ac38855b1c083d4f on tablet b9431cf5853c4148ac38855b1c083d4f with peer ca177138f2c44d2eafa1646b795f7608
I20240501 07:59:56.869870 14659 tablet_bootstrap.cc:492] T f61e70ba02384f81b9ae36b1c65465b9 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:56.870188 14659 tablet_bootstrap.cc:492] T f61e70ba02384f81b9ae36b1c65465b9 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap complete.
I20240501 07:59:56.870411 14659 ts_tablet_manager.cc:1367] T f61e70ba02384f81b9ae36b1c65465b9 P e5c03261d31c4d9a91077a17e1cd0336: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.000s
I20240501 07:59:56.870973 14659 raft_consensus.cc:348] T f61e70ba02384f81b9ae36b1c65465b9 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:56.871106 14659 raft_consensus.cc:729] T f61e70ba02384f81b9ae36b1c65465b9 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c03261d31c4d9a91077a17e1cd0336, State: Initialized, Role: FOLLOWER
I20240501 07:59:56.871289 14659 consensus_queue.cc:260] T f61e70ba02384f81b9ae36b1c65465b9 P e5c03261d31c4d9a91077a17e1cd0336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:56.872071 14659 ts_tablet_manager.cc:1398] T f61e70ba02384f81b9ae36b1c65465b9 P e5c03261d31c4d9a91077a17e1cd0336: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:56.872572 11683 tablet_copy_service.cc:342] P 603919bb02364c04b6c530245b8b02f2: Request end of tablet copy session e5c03261d31c4d9a91077a17e1cd0336-f61e70ba02384f81b9ae36b1c65465b9 received from {username='slave'} at 127.0.99.135:57735
I20240501 07:59:56.872655 11683 tablet_copy_service.cc:433] P 603919bb02364c04b6c530245b8b02f2: ending tablet copy session e5c03261d31c4d9a91077a17e1cd0336-f61e70ba02384f81b9ae36b1c65465b9 on tablet f61e70ba02384f81b9ae36b1c65465b9 with peer e5c03261d31c4d9a91077a17e1cd0336
I20240501 07:59:56.875624 14893 tablet_bootstrap.cc:492] T 64613ea537c842f29d810e829278394e P 60e8857abe7c41038af46ebb5a095a90: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:56.875933 14893 tablet_bootstrap.cc:492] T 64613ea537c842f29d810e829278394e P 60e8857abe7c41038af46ebb5a095a90: Bootstrap complete.
I20240501 07:59:56.876106 14893 ts_tablet_manager.cc:1367] T 64613ea537c842f29d810e829278394e P 60e8857abe7c41038af46ebb5a095a90: Time spent bootstrapping tablet: real 0.314s user 0.022s sys 0.009s
I20240501 07:59:56.879317 14560 ts_tablet_manager.cc:902] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9: Initiating tablet copy from peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:56.879781 14560 tablet_copy_client.cc:308] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:44403
I20240501 07:59:56.880733 11683 tablet_copy_service.cc:140] P 603919bb02364c04b6c530245b8b02f2: Received BeginTabletCopySession request for tablet 5b7de08d2f004cd8abc2bbe1cf7b1444 from peer 53d1d2c0767b4f6195b69758a3dd6cf9 ({username='slave'} at 127.0.99.134:57705)
I20240501 07:59:56.880867 11683 tablet_copy_service.cc:161] P 603919bb02364c04b6c530245b8b02f2: Beginning new tablet copy session on tablet 5b7de08d2f004cd8abc2bbe1cf7b1444 from peer 53d1d2c0767b4f6195b69758a3dd6cf9 at {username='slave'} at 127.0.99.134:57705: session id = 53d1d2c0767b4f6195b69758a3dd6cf9-5b7de08d2f004cd8abc2bbe1cf7b1444
I20240501 07:59:56.881788 11683 tablet_copy_source_session.cc:205] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:56.882899 14560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7de08d2f004cd8abc2bbe1cf7b1444. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:56.886585 14560 tablet_copy_client.cc:783] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:56.886775 14560 tablet_copy_client.cc:647] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:56.889283 14893 raft_consensus.cc:348] T 64613ea537c842f29d810e829278394e P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:56.889457 14893 raft_consensus.cc:729] T 64613ea537c842f29d810e829278394e P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e8857abe7c41038af46ebb5a095a90, State: Initialized, Role: FOLLOWER
I20240501 07:59:56.889612 14893 consensus_queue.cc:260] T 64613ea537c842f29d810e829278394e P 60e8857abe7c41038af46ebb5a095a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:56.889916 14893 ts_tablet_manager.cc:1398] T 64613ea537c842f29d810e829278394e P 60e8857abe7c41038af46ebb5a095a90: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 07:59:56.891633 11420 tablet_copy_service.cc:342] P 655ebda1fb2c49c4af7087a760529835: Request end of tablet copy session 60e8857abe7c41038af46ebb5a095a90-64613ea537c842f29d810e829278394e received from {username='slave'} at 127.0.99.133:38419
I20240501 07:59:56.891738 11420 tablet_copy_service.cc:433] P 655ebda1fb2c49c4af7087a760529835: ending tablet copy session 60e8857abe7c41038af46ebb5a095a90-64613ea537c842f29d810e829278394e on tablet 64613ea537c842f29d810e829278394e with peer 60e8857abe7c41038af46ebb5a095a90
I20240501 07:59:56.891970 14988 tablet_bootstrap.cc:492] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:56.892259 14988 tablet_bootstrap.cc:492] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608: Bootstrap complete.
I20240501 07:59:56.892449 14988 ts_tablet_manager.cc:1367] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608: Time spent bootstrapping tablet: real 0.068s user 0.025s sys 0.000s
I20240501 07:59:56.892472 13317 raft_consensus.cc:1203] T 18b146cc4e164e1d9f9f827217c9b3ab P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:56.892911 14988 raft_consensus.cc:348] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:56.893043 14988 raft_consensus.cc:729] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca177138f2c44d2eafa1646b795f7608, State: Initialized, Role: FOLLOWER
I20240501 07:59:56.893391 14988 consensus_queue.cc:260] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:56.894115 14988 ts_tablet_manager.cc:1398] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:56.894762 11290 tablet_copy_service.cc:342] P e2dafc1866314669ad0de8fb7bf7fd05: Request end of tablet copy session ca177138f2c44d2eafa1646b795f7608-867ba08a15b44d4c9e6fa92b0d518d1d received from {username='slave'} at 127.0.99.136:55185
I20240501 07:59:56.894847 11290 tablet_copy_service.cc:433] P e2dafc1866314669ad0de8fb7bf7fd05: ending tablet copy session ca177138f2c44d2eafa1646b795f7608-867ba08a15b44d4c9e6fa92b0d518d1d on tablet 867ba08a15b44d4c9e6fa92b0d518d1d with peer ca177138f2c44d2eafa1646b795f7608
I20240501 07:59:56.895466 11532 consensus_queue.cc:237] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
W20240501 07:59:56.897643 11465 consensus_peers.cc:485] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771): Couldn't send request to peer 53d1d2c0767b4f6195b69758a3dd6cf9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4d15e19eed54aefb6b96276fad98324. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:56.897739 11269 raft_consensus.cc:1261] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.898077 11401 raft_consensus.cc:1261] T b4d15e19eed54aefb6b96276fad98324 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.898468 14338 consensus_queue.cc:1035] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.898859 14338 consensus_queue.cc:1035] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.900760 14918 tablet_bootstrap.cc:492] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:56.900939 14540 tablet_bootstrap.cc:492] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:56.901051 14918 tablet_bootstrap.cc:492] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90: Bootstrap complete.
I20240501 07:59:56.901203 14540 tablet_bootstrap.cc:492] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608: Bootstrap complete.
I20240501 07:59:56.901265 14918 ts_tablet_manager.cc:1367] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90: Time spent bootstrapping tablet: real 0.107s user 0.028s sys 0.000s
I20240501 07:59:56.901376 14540 ts_tablet_manager.cc:1367] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608: Time spent bootstrapping tablet: real 0.155s user 0.026s sys 0.000s
I20240501 07:59:56.901834 14540 raft_consensus.cc:348] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:56.901952 14540 raft_consensus.cc:729] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca177138f2c44d2eafa1646b795f7608, State: Initialized, Role: FOLLOWER
I20240501 07:59:56.902542 14540 consensus_queue.cc:260] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:56.903856 14540 ts_tablet_manager.cc:1398] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20240501 07:59:56.904304 11420 tablet_copy_service.cc:342] P 655ebda1fb2c49c4af7087a760529835: Request end of tablet copy session ca177138f2c44d2eafa1646b795f7608-90f11b5693d7456d8a604e6c8726e69a received from {username='slave'} at 127.0.99.136:50157
I20240501 07:59:56.904399 11420 tablet_copy_service.cc:433] P 655ebda1fb2c49c4af7087a760529835: ending tablet copy session ca177138f2c44d2eafa1646b795f7608-90f11b5693d7456d8a604e6c8726e69a on tablet 90f11b5693d7456d8a604e6c8726e69a with peer ca177138f2c44d2eafa1646b795f7608
I20240501 07:59:56.904960 14560 tablet_copy_client.cc:523] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:56.907184 14918 raft_consensus.cc:348] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:56.907349 14918 raft_consensus.cc:729] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e8857abe7c41038af46ebb5a095a90, State: Initialized, Role: FOLLOWER
I20240501 07:59:56.907485 14918 consensus_queue.cc:260] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:56.907775 14918 ts_tablet_manager.cc:1398] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:56.907884 14560 tablet_bootstrap.cc:492] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap starting.
I20240501 07:59:56.908425 11420 tablet_copy_service.cc:342] P 655ebda1fb2c49c4af7087a760529835: Request end of tablet copy session 60e8857abe7c41038af46ebb5a095a90-5e8550ea95984d08937b9b3e2af12bec received from {username='slave'} at 127.0.99.133:38419
I20240501 07:59:56.908500 11420 tablet_copy_service.cc:433] P 655ebda1fb2c49c4af7087a760529835: ending tablet copy session 60e8857abe7c41038af46ebb5a095a90-5e8550ea95984d08937b9b3e2af12bec on tablet 5e8550ea95984d08937b9b3e2af12bec with peer 60e8857abe7c41038af46ebb5a095a90
I20240501 07:59:56.909530 11532 consensus_queue.cc:237] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:56.914247 13054 raft_consensus.cc:1261] T 6d214be86534458091f54a386d7d3334 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:56.914934 11270 raft_consensus.cc:1261] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:56.915441 14350 consensus_queue.cc:1035] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:56.915782 11662 raft_consensus.cc:1261] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:56.916147 14338 consensus_queue.cc:1035] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:56.916597 14338 consensus_queue.cc:1035] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:56.917148 14958 tablet_bootstrap.cc:492] T 164fd265561e47128e7ff585b8cfab07 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:56.917479 14958 tablet_bootstrap.cc:492] T 164fd265561e47128e7ff585b8cfab07 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap complete.
I20240501 07:59:56.917696 14958 ts_tablet_manager.cc:1367] T 164fd265561e47128e7ff585b8cfab07 P 60e8857abe7c41038af46ebb5a095a90: Time spent bootstrapping tablet: real 0.105s user 0.028s sys 0.001s
I20240501 07:59:56.918179 14958 raft_consensus.cc:348] T 164fd265561e47128e7ff585b8cfab07 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:56.918331 14958 raft_consensus.cc:729] T 164fd265561e47128e7ff585b8cfab07 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e8857abe7c41038af46ebb5a095a90, State: Initialized, Role: FOLLOWER
I20240501 07:59:56.918514 14958 consensus_queue.cc:260] T 164fd265561e47128e7ff585b8cfab07 P 60e8857abe7c41038af46ebb5a095a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:56.918833 14958 ts_tablet_manager.cc:1398] T 164fd265561e47128e7ff585b8cfab07 P 60e8857abe7c41038af46ebb5a095a90: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:56.919332 11420 tablet_copy_service.cc:342] P 655ebda1fb2c49c4af7087a760529835: Request end of tablet copy session 60e8857abe7c41038af46ebb5a095a90-164fd265561e47128e7ff585b8cfab07 received from {username='slave'} at 127.0.99.133:38419
I20240501 07:59:56.919420 11420 tablet_copy_service.cc:433] P 655ebda1fb2c49c4af7087a760529835: ending tablet copy session 60e8857abe7c41038af46ebb5a095a90-164fd265561e47128e7ff585b8cfab07 on tablet 164fd265561e47128e7ff585b8cfab07 with peer 60e8857abe7c41038af46ebb5a095a90
I20240501 07:59:56.919843 14338 raft_consensus.cc:2932] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:56.923560 11270 raft_consensus.cc:2932] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:56.926900 11128 catalog_manager.cc:5484] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 4 to 5, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:56.927493 11662 raft_consensus.cc:2932] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:56.927736 11400 consensus_queue.cc:237] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
W20240501 07:59:56.930372 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1ebf29682024645978b3caa2d4988f3. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:56.931398 11269 raft_consensus.cc:1261] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.931844 14456 consensus_queue.cc:1035] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:56.932673 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d214be86534458091f54a386d7d3334. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:56.933584 11662 raft_consensus.cc:1261] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:56.933883 15081 raft_consensus.cc:2932] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:56.934211 14456 consensus_queue.cc:1035] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:56.935693 11401 raft_consensus.cc:2932] T b4d15e19eed54aefb6b96276fad98324 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:56.937527 13054 raft_consensus.cc:2932] T 6d214be86534458091f54a386d7d3334 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:56.940603 11270 raft_consensus.cc:2932] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:56.940979 11128 catalog_manager.cc:5484] T b4d15e19eed54aefb6b96276fad98324 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:56.949126 15107 raft_consensus.cc:2932] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:56.949549 11269 raft_consensus.cc:2932] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:56.951493 11662 raft_consensus.cc:2932] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:56.952499 11128 catalog_manager.cc:5484] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:56.956347 14560 tablet_bootstrap.cc:492] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:56.956598 13185 raft_consensus.cc:1203] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:56.956640 14560 tablet_bootstrap.cc:492] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap complete.
I20240501 07:59:56.956826 14560 ts_tablet_manager.cc:1367] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent bootstrapping tablet: real 0.049s user 0.023s sys 0.000s
I20240501 07:59:56.957520 14560 raft_consensus.cc:348] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:56.957653 14560 raft_consensus.cc:729] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d1d2c0767b4f6195b69758a3dd6cf9, State: Initialized, Role: FOLLOWER
I20240501 07:59:56.957775 14560 consensus_queue.cc:260] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:56.958082 14560 ts_tablet_manager.cc:1398] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 07:59:56.960049 11683 tablet_copy_service.cc:342] P 603919bb02364c04b6c530245b8b02f2: Request end of tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-5b7de08d2f004cd8abc2bbe1cf7b1444 received from {username='slave'} at 127.0.99.134:57705
I20240501 07:59:56.960139 11683 tablet_copy_service.cc:433] P 603919bb02364c04b6c530245b8b02f2: ending tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-5b7de08d2f004cd8abc2bbe1cf7b1444 on tablet 5b7de08d2f004cd8abc2bbe1cf7b1444 with peer 53d1d2c0767b4f6195b69758a3dd6cf9
I20240501 07:59:56.983637 11662 consensus_queue.cc:237] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:56.985723 11401 raft_consensus.cc:1261] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:56.985749 11532 raft_consensus.cc:1261] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:56.985819 13317 raft_consensus.cc:1261] T f61e70ba02384f81b9ae36b1c65465b9 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:56.986200 14641 consensus_queue.cc:1035] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:56.986347 13507 consensus_queue.cc:1035] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:56.986718 14960 consensus_queue.cc:1035] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 07:59:56.986869 11596 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771): Couldn't send request to peer 53d1d2c0767b4f6195b69758a3dd6cf9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f61e70ba02384f81b9ae36b1c65465b9. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:56.990298 14960 raft_consensus.cc:2932] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:56.991519 11401 consensus_queue.cc:237] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:56.992676 11401 raft_consensus.cc:2932] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:56.993142 11532 raft_consensus.cc:2932] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:56.993610 13054 raft_consensus.cc:1261] T 164fd265561e47128e7ff585b8cfab07 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:56.994521 14430 consensus_queue.cc:1035] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:56.994714 11530 raft_consensus.cc:1261] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:56.994971 11269 raft_consensus.cc:1261] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:56.996013 14456 consensus_queue.cc:1035] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:56.996194 11120 catalog_manager.cc:5484] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 4 to 5, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:56.997812 13317 raft_consensus.cc:2932] T f61e70ba02384f81b9ae36b1c65465b9 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
W20240501 07:59:56.998580 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 164fd265561e47128e7ff585b8cfab07. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:57.000290 14430 consensus_queue.cc:1035] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:57.003396 14456 raft_consensus.cc:2932] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.004884 13054 raft_consensus.cc:2932] T 164fd265561e47128e7ff585b8cfab07 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.007190 11530 raft_consensus.cc:2932] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.007385 11125 catalog_manager.cc:5484] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 4 to 5, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.009625 11269 raft_consensus.cc:2932] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.017020 11269 consensus_queue.cc:237] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:57.018633 13054 raft_consensus.cc:1203] T 64613ea537c842f29d810e829278394e P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 07:59:57.019213 11401 raft_consensus.cc:1261] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:57.019707 14290 consensus_queue.cc:1035] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:57.020220 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c95c4f3ef08f44d1b69fb7b98426f5af. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:57.021931 11530 raft_consensus.cc:1261] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:57.022405 14290 consensus_queue.cc:1035] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:57.024996 15026 raft_consensus.cc:2932] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:57.025662 11530 raft_consensus.cc:2932] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:57.027271 11401 raft_consensus.cc:2932] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:57.028484 11128 catalog_manager.cc:5484] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:57.059154 11401 consensus_queue.cc:237] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:57.061196 11530 raft_consensus.cc:1261] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:57.061556 13054 raft_consensus.cc:1261] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:57.061594 13053 raft_consensus.cc:1203] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 07:59:57.062265 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f6e4ae97e884219ac7d0d94997d9cc9. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:57.062608 14430 consensus_queue.cc:1035] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:57.062726 11662 raft_consensus.cc:1261] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:57.063220 14456 consensus_queue.cc:1035] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:57.064684 13317 raft_consensus.cc:1203] T 64613ea537c842f29d810e829278394e P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 07:59:57.066402 14456 consensus_queue.cc:1035] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:57.067929 15127 raft_consensus.cc:2932] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.069032 11662 raft_consensus.cc:2932] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.072162 11121 catalog_manager.cc:5484] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2 reported cstate change: config changed from index 4 to 5, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.072739 11530 raft_consensus.cc:2932] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.074380 14958 ts_tablet_manager.cc:902] T ed88ea311f474ab28cc6fbe6069258bf P 60e8857abe7c41038af46ebb5a095a90: Initiating tablet copy from peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:57.074923 14958 tablet_copy_client.cc:308] T ed88ea311f474ab28cc6fbe6069258bf P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:44403
I20240501 07:59:57.074873 13054 raft_consensus.cc:2932] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.083029 11683 tablet_copy_service.cc:140] P 603919bb02364c04b6c530245b8b02f2: Received BeginTabletCopySession request for tablet ed88ea311f474ab28cc6fbe6069258bf from peer 60e8857abe7c41038af46ebb5a095a90 ({username='slave'} at 127.0.99.133:51087)
I20240501 07:59:57.083170 11683 tablet_copy_service.cc:161] P 603919bb02364c04b6c530245b8b02f2: Beginning new tablet copy session on tablet ed88ea311f474ab28cc6fbe6069258bf from peer 60e8857abe7c41038af46ebb5a095a90 at {username='slave'} at 127.0.99.133:51087: session id = 60e8857abe7c41038af46ebb5a095a90-ed88ea311f474ab28cc6fbe6069258bf
I20240501 07:59:57.084101 11683 tablet_copy_source_session.cc:205] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:57.085060 14958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed88ea311f474ab28cc6fbe6069258bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:57.087491 14958 tablet_copy_client.cc:783] T ed88ea311f474ab28cc6fbe6069258bf P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:57.087651 14958 tablet_copy_client.cc:647] T ed88ea311f474ab28cc6fbe6069258bf P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:57.112890 14958 tablet_copy_client.cc:523] T ed88ea311f474ab28cc6fbe6069258bf P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:57.113684 11401 consensus_queue.cc:237] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:57.114290 14958 tablet_bootstrap.cc:492] T ed88ea311f474ab28cc6fbe6069258bf P 60e8857abe7c41038af46ebb5a095a90: Bootstrap starting.
I20240501 07:59:57.115803 11269 raft_consensus.cc:1261] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:57.116458 11662 raft_consensus.cc:1261] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:57.116892 14456 consensus_queue.cc:1035] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:57.117836 14918 ts_tablet_manager.cc:902] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90: Initiating tablet copy from peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 07:59:57.118041 14456 consensus_queue.cc:1035] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:57.118297 14918 tablet_copy_client.cc:308] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46085
W20240501 07:59:57.118489 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1ebf29682024645978b3caa2d4988f3. This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:57.119511 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1ebf29682024645978b3caa2d4988f3. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:57.121496 14456 raft_consensus.cc:2932] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:57.124246 11269 raft_consensus.cc:2932] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:57.124418 11290 tablet_copy_service.cc:140] P e2dafc1866314669ad0de8fb7bf7fd05: Received BeginTabletCopySession request for tablet 25cee7e51c7c438aa721d71fb82dc71e from peer 60e8857abe7c41038af46ebb5a095a90 ({username='slave'} at 127.0.99.133:60295)
I20240501 07:59:57.124512 11290 tablet_copy_service.cc:161] P e2dafc1866314669ad0de8fb7bf7fd05: Beginning new tablet copy session on tablet 25cee7e51c7c438aa721d71fb82dc71e from peer 60e8857abe7c41038af46ebb5a095a90 at {username='slave'} at 127.0.99.133:60295: session id = 60e8857abe7c41038af46ebb5a095a90-25cee7e51c7c438aa721d71fb82dc71e
I20240501 07:59:57.125450 11290 tablet_copy_source_session.cc:205] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:57.127375 14918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25cee7e51c7c438aa721d71fb82dc71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:57.127638 11662 raft_consensus.cc:2932] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:57.128408 11121 catalog_manager.cc:5484] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 4 to 5, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:57.129608 14918 tablet_copy_client.cc:783] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:57.129760 14918 tablet_copy_client.cc:647] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:57.144008 14918 tablet_copy_client.cc:523] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:57.145689 14918 tablet_bootstrap.cc:492] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90: Bootstrap starting.
I20240501 07:59:57.154495 11662 consensus_queue.cc:237] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:57.154706 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:57.157786 11401 raft_consensus.cc:1261] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:57.158121 11269 raft_consensus.cc:1261] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:57.158883 13507 consensus_queue.cc:1035] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:57.160065 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.163937 13507 consensus_queue.cc:1035] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:57.167209 13449 raft_consensus.cc:1203] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:57.168022 14958 tablet_bootstrap.cc:492] T ed88ea311f474ab28cc6fbe6069258bf P 60e8857abe7c41038af46ebb5a095a90: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:57.168378 14958 tablet_bootstrap.cc:492] T ed88ea311f474ab28cc6fbe6069258bf P 60e8857abe7c41038af46ebb5a095a90: Bootstrap complete.
I20240501 07:59:57.168581 14958 ts_tablet_manager.cc:1367] T ed88ea311f474ab28cc6fbe6069258bf P 60e8857abe7c41038af46ebb5a095a90: Time spent bootstrapping tablet: real 0.054s user 0.026s sys 0.000s
I20240501 07:59:57.169060 14958 raft_consensus.cc:348] T ed88ea311f474ab28cc6fbe6069258bf P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:57.169189 14958 raft_consensus.cc:729] T ed88ea311f474ab28cc6fbe6069258bf P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e8857abe7c41038af46ebb5a095a90, State: Initialized, Role: FOLLOWER
I20240501 07:59:57.170941 13449 raft_consensus.cc:1203] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:57.172777 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:57.177031 14958 consensus_queue.cc:260] T ed88ea311f474ab28cc6fbe6069258bf P 60e8857abe7c41038af46ebb5a095a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:57.177520 14958 ts_tablet_manager.cc:1398] T ed88ea311f474ab28cc6fbe6069258bf P 60e8857abe7c41038af46ebb5a095a90: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240501 07:59:57.178046 11683 tablet_copy_service.cc:342] P 603919bb02364c04b6c530245b8b02f2: Request end of tablet copy session 60e8857abe7c41038af46ebb5a095a90-ed88ea311f474ab28cc6fbe6069258bf received from {username='slave'} at 127.0.99.133:51087
I20240501 07:59:57.178143 11683 tablet_copy_service.cc:433] P 603919bb02364c04b6c530245b8b02f2: ending tablet copy session 60e8857abe7c41038af46ebb5a095a90-ed88ea311f474ab28cc6fbe6069258bf on tablet ed88ea311f474ab28cc6fbe6069258bf with peer 60e8857abe7c41038af46ebb5a095a90
I20240501 07:59:57.178813 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:57.189436 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.195135 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:57.204468 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:57.205890 14918 tablet_bootstrap.cc:492] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:57.206248 14918 tablet_bootstrap.cc:492] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90: Bootstrap complete.
I20240501 07:59:57.206526 14918 ts_tablet_manager.cc:1367] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90: Time spent bootstrapping tablet: real 0.061s user 0.027s sys 0.002s
I20240501 07:59:57.207068 14918 raft_consensus.cc:348] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:57.207296 14918 raft_consensus.cc:729] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e8857abe7c41038af46ebb5a095a90, State: Initialized, Role: FOLLOWER
I20240501 07:59:57.207504 14918 consensus_queue.cc:260] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:57.207896 14918 ts_tablet_manager.cc:1398] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:57.208365 11290 tablet_copy_service.cc:342] P e2dafc1866314669ad0de8fb7bf7fd05: Request end of tablet copy session 60e8857abe7c41038af46ebb5a095a90-25cee7e51c7c438aa721d71fb82dc71e received from {username='slave'} at 127.0.99.133:60295
I20240501 07:59:57.208447 11290 tablet_copy_service.cc:433] P e2dafc1866314669ad0de8fb7bf7fd05: ending tablet copy session 60e8857abe7c41038af46ebb5a095a90-25cee7e51c7c438aa721d71fb82dc71e on tablet 25cee7e51c7c438aa721d71fb82dc71e with peer 60e8857abe7c41038af46ebb5a095a90
I20240501 07:59:57.213197 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.225455 13449 raft_consensus.cc:1203] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:57.235433 13507 raft_consensus.cc:2932] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
W20240501 07:59:57.236460 11599 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53193305f868433c852755e8a8a81766. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:57.237640 11401 raft_consensus.cc:2932] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.238297 11122 catalog_manager.cc:5484] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 reported cstate change: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.238632 11269 raft_consensus.cc:2932] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.242003 14560 ts_tablet_manager.cc:902] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9: Initiating tablet copy from peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 07:59:57.242506 14560 tablet_copy_client.cc:308] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46085
I20240501 07:59:57.243396 11290 tablet_copy_service.cc:140] P e2dafc1866314669ad0de8fb7bf7fd05: Received BeginTabletCopySession request for tablet ea97f34590684bcf998228245dbcd9b0 from peer 53d1d2c0767b4f6195b69758a3dd6cf9 ({username='slave'} at 127.0.99.134:37773)
I20240501 07:59:57.243499 11290 tablet_copy_service.cc:161] P e2dafc1866314669ad0de8fb7bf7fd05: Beginning new tablet copy session on tablet ea97f34590684bcf998228245dbcd9b0 from peer 53d1d2c0767b4f6195b69758a3dd6cf9 at {username='slave'} at 127.0.99.134:37773: session id = 53d1d2c0767b4f6195b69758a3dd6cf9-ea97f34590684bcf998228245dbcd9b0
I20240501 07:59:57.244484 11290 tablet_copy_source_session.cc:205] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:57.245759 14560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea97f34590684bcf998228245dbcd9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:57.248106 14560 tablet_copy_client.cc:783] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:57.248257 14560 tablet_copy_client.cc:647] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:57.249516 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.261770 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.263617 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.265163 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:57.265195 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.277294 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.277506 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:57.302956 14560 tablet_copy_client.cc:523] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:57.304749 14560 tablet_bootstrap.cc:492] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap starting.
I20240501 07:59:57.306335 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:57.314826 13185 raft_consensus.cc:1203] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:57.336637 14560 tablet_bootstrap.cc:492] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:57.337054 14560 tablet_bootstrap.cc:492] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap complete.
I20240501 07:59:57.337283 14560 ts_tablet_manager.cc:1367] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent bootstrapping tablet: real 0.033s user 0.029s sys 0.000s
I20240501 07:59:57.337803 14560 raft_consensus.cc:348] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:57.337998 14560 raft_consensus.cc:729] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d1d2c0767b4f6195b69758a3dd6cf9, State: Initialized, Role: FOLLOWER
I20240501 07:59:57.338234 14560 consensus_queue.cc:260] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:57.338862 14560 ts_tablet_manager.cc:1398] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:57.339444 11290 tablet_copy_service.cc:342] P e2dafc1866314669ad0de8fb7bf7fd05: Request end of tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-ea97f34590684bcf998228245dbcd9b0 received from {username='slave'} at 127.0.99.134:37773
I20240501 07:59:57.339540 11290 tablet_copy_service.cc:433] P e2dafc1866314669ad0de8fb7bf7fd05: ending tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-ea97f34590684bcf998228245dbcd9b0 on tablet ea97f34590684bcf998228245dbcd9b0 with peer 53d1d2c0767b4f6195b69758a3dd6cf9
I20240501 07:59:57.342063 14659 ts_tablet_manager.cc:902] T 6d214be86534458091f54a386d7d3334 P e5c03261d31c4d9a91077a17e1cd0336: Initiating tablet copy from peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:57.342555 14659 tablet_copy_client.cc:308] T 6d214be86534458091f54a386d7d3334 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37957
I20240501 07:59:57.349418 11552 tablet_copy_service.cc:140] P 22ec12774a4047a8b859743dc73f8ff6: Received BeginTabletCopySession request for tablet 6d214be86534458091f54a386d7d3334 from peer e5c03261d31c4d9a91077a17e1cd0336 ({username='slave'} at 127.0.99.135:47673)
I20240501 07:59:57.349552 11552 tablet_copy_service.cc:161] P 22ec12774a4047a8b859743dc73f8ff6: Beginning new tablet copy session on tablet 6d214be86534458091f54a386d7d3334 from peer e5c03261d31c4d9a91077a17e1cd0336 at {username='slave'} at 127.0.99.135:47673: session id = e5c03261d31c4d9a91077a17e1cd0336-6d214be86534458091f54a386d7d3334
I20240501 07:59:57.350607 11552 tablet_copy_source_session.cc:205] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:57.351739 14659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d214be86534458091f54a386d7d3334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:57.354161 14659 tablet_copy_client.cc:783] T 6d214be86534458091f54a386d7d3334 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:57.354334 14659 tablet_copy_client.cc:647] T 6d214be86534458091f54a386d7d3334 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:57.359683 14659 tablet_copy_client.cc:523] T 6d214be86534458091f54a386d7d3334 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:57.361661 14659 tablet_bootstrap.cc:492] T 6d214be86534458091f54a386d7d3334 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap starting.
I20240501 07:59:57.371173 11530 consensus_queue.cc:237] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:57.373212 11401 raft_consensus.cc:1261] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:57.373212 11662 raft_consensus.cc:1261] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:57.374236 11465 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771): Couldn't send request to peer 53d1d2c0767b4f6195b69758a3dd6cf9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48b0ff0769dc426190f038811155c741. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:57.374364 14338 consensus_queue.cc:1035] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:57.374780 14338 consensus_queue.cc:1035] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:57.378196 14350 raft_consensus.cc:2932] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:57.380533 11401 raft_consensus.cc:2932] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:57.380892 11122 catalog_manager.cc:5484] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:57.384068 11662 raft_consensus.cc:2932] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:57.392385 14659 tablet_bootstrap.cc:492] T 6d214be86534458091f54a386d7d3334 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:57.392709 14659 tablet_bootstrap.cc:492] T 6d214be86534458091f54a386d7d3334 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap complete.
I20240501 07:59:57.392922 14659 ts_tablet_manager.cc:1367] T 6d214be86534458091f54a386d7d3334 P e5c03261d31c4d9a91077a17e1cd0336: Time spent bootstrapping tablet: real 0.031s user 0.023s sys 0.000s
I20240501 07:59:57.393462 14659 raft_consensus.cc:348] T 6d214be86534458091f54a386d7d3334 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:57.393595 14659 raft_consensus.cc:729] T 6d214be86534458091f54a386d7d3334 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c03261d31c4d9a91077a17e1cd0336, State: Initialized, Role: FOLLOWER
I20240501 07:59:57.393744 14659 consensus_queue.cc:260] T 6d214be86534458091f54a386d7d3334 P e5c03261d31c4d9a91077a17e1cd0336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:57.397195 14659 ts_tablet_manager.cc:1398] T 6d214be86534458091f54a386d7d3334 P e5c03261d31c4d9a91077a17e1cd0336: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 07:59:57.398756 11552 tablet_copy_service.cc:342] P 22ec12774a4047a8b859743dc73f8ff6: Request end of tablet copy session e5c03261d31c4d9a91077a17e1cd0336-6d214be86534458091f54a386d7d3334 received from {username='slave'} at 127.0.99.135:47673
I20240501 07:59:57.398864 11552 tablet_copy_service.cc:433] P 22ec12774a4047a8b859743dc73f8ff6: ending tablet copy session e5c03261d31c4d9a91077a17e1cd0336-6d214be86534458091f54a386d7d3334 on tablet 6d214be86534458091f54a386d7d3334 with peer e5c03261d31c4d9a91077a17e1cd0336
I20240501 07:59:57.399220 14560 ts_tablet_manager.cc:902] T b4d15e19eed54aefb6b96276fad98324 P 53d1d2c0767b4f6195b69758a3dd6cf9: Initiating tablet copy from peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:57.399654 14560 tablet_copy_client.cc:308] T b4d15e19eed54aefb6b96276fad98324 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37957
I20240501 07:59:57.400317 11552 tablet_copy_service.cc:140] P 22ec12774a4047a8b859743dc73f8ff6: Received BeginTabletCopySession request for tablet b4d15e19eed54aefb6b96276fad98324 from peer 53d1d2c0767b4f6195b69758a3dd6cf9 ({username='slave'} at 127.0.99.134:40967)
I20240501 07:59:57.400430 11552 tablet_copy_service.cc:161] P 22ec12774a4047a8b859743dc73f8ff6: Beginning new tablet copy session on tablet b4d15e19eed54aefb6b96276fad98324 from peer 53d1d2c0767b4f6195b69758a3dd6cf9 at {username='slave'} at 127.0.99.134:40967: session id = 53d1d2c0767b4f6195b69758a3dd6cf9-b4d15e19eed54aefb6b96276fad98324
I20240501 07:59:57.401446 11552 tablet_copy_source_session.cc:205] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:57.402405 14560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4d15e19eed54aefb6b96276fad98324. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:57.404795 14560 tablet_copy_client.cc:783] T b4d15e19eed54aefb6b96276fad98324 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:57.404953 14560 tablet_copy_client.cc:647] T b4d15e19eed54aefb6b96276fad98324 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:57.410216 14560 tablet_copy_client.cc:523] T b4d15e19eed54aefb6b96276fad98324 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:57.412330 14560 tablet_bootstrap.cc:492] T b4d15e19eed54aefb6b96276fad98324 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap starting.
I20240501 07:59:57.420727 15402 ts_tablet_manager.cc:902] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608: Initiating tablet copy from peer 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:57.421478 15402 tablet_copy_client.cc:308] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41575
I20240501 07:59:57.422518 11420 tablet_copy_service.cc:140] P 655ebda1fb2c49c4af7087a760529835: Received BeginTabletCopySession request for tablet 164fd265561e47128e7ff585b8cfab07 from peer ca177138f2c44d2eafa1646b795f7608 ({username='slave'} at 127.0.99.136:50157)
I20240501 07:59:57.422629 11420 tablet_copy_service.cc:161] P 655ebda1fb2c49c4af7087a760529835: Beginning new tablet copy session on tablet 164fd265561e47128e7ff585b8cfab07 from peer ca177138f2c44d2eafa1646b795f7608 at {username='slave'} at 127.0.99.136:50157: session id = ca177138f2c44d2eafa1646b795f7608-164fd265561e47128e7ff585b8cfab07
I20240501 07:59:57.423517 11420 tablet_copy_source_session.cc:205] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:57.426118 15402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164fd265561e47128e7ff585b8cfab07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:57.426023 11401 consensus_queue.cc:237] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:57.427702 11530 raft_consensus.cc:1261] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:57.428277 11662 raft_consensus.cc:1261] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:57.429611 15107 consensus_queue.cc:1035] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:57.431318 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c23e3bd7ec34876956046e5d32e4d82. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:57.431403 15402 tablet_copy_client.cc:783] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:57.431583 15402 tablet_copy_client.cc:647] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:57.431849 15107 consensus_queue.cc:1035] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:57.433657 15402 tablet_copy_client.cc:523] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:57.447654 14560 tablet_bootstrap.cc:492] T b4d15e19eed54aefb6b96276fad98324 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:57.447952 14560 tablet_bootstrap.cc:492] T b4d15e19eed54aefb6b96276fad98324 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap complete.
I20240501 07:59:57.448139 14560 ts_tablet_manager.cc:1367] T b4d15e19eed54aefb6b96276fad98324 P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent bootstrapping tablet: real 0.036s user 0.019s sys 0.007s
I20240501 07:59:57.448640 14560 raft_consensus.cc:348] T b4d15e19eed54aefb6b96276fad98324 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:57.448766 14560 raft_consensus.cc:729] T b4d15e19eed54aefb6b96276fad98324 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d1d2c0767b4f6195b69758a3dd6cf9, State: Initialized, Role: FOLLOWER
I20240501 07:59:57.448884 14560 consensus_queue.cc:260] T b4d15e19eed54aefb6b96276fad98324 P 53d1d2c0767b4f6195b69758a3dd6cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:57.449138 14560 ts_tablet_manager.cc:1398] T b4d15e19eed54aefb6b96276fad98324 P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:57.450260 11552 tablet_copy_service.cc:342] P 22ec12774a4047a8b859743dc73f8ff6: Request end of tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-b4d15e19eed54aefb6b96276fad98324 received from {username='slave'} at 127.0.99.134:40967
I20240501 07:59:57.450354 11552 tablet_copy_service.cc:433] P 22ec12774a4047a8b859743dc73f8ff6: ending tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-b4d15e19eed54aefb6b96276fad98324 on tablet b4d15e19eed54aefb6b96276fad98324 with peer 53d1d2c0767b4f6195b69758a3dd6cf9
I20240501 07:59:57.454124 15402 tablet_bootstrap.cc:492] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608: Bootstrap starting.
I20240501 07:59:57.461385 15236 raft_consensus.cc:2932] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:57.461962 11530 raft_consensus.cc:2932] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:57.468364 11662 raft_consensus.cc:2932] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:57.468827 11126 catalog_manager.cc:5484] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index -1 to 4, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:57.491283 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:57.497057 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:57.499825 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.514568 14918 ts_tablet_manager.cc:902] T a1ebf29682024645978b3caa2d4988f3 P 60e8857abe7c41038af46ebb5a095a90: Initiating tablet copy from peer 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:57.515084 14918 tablet_copy_client.cc:308] T a1ebf29682024645978b3caa2d4988f3 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41575
I20240501 07:59:57.515491 15402 tablet_bootstrap.cc:492] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:57.515859 15402 tablet_bootstrap.cc:492] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608: Bootstrap complete.
I20240501 07:59:57.515843 11420 tablet_copy_service.cc:140] P 655ebda1fb2c49c4af7087a760529835: Received BeginTabletCopySession request for tablet a1ebf29682024645978b3caa2d4988f3 from peer 60e8857abe7c41038af46ebb5a095a90 ({username='slave'} at 127.0.99.133:38419)
I20240501 07:59:57.515954 11420 tablet_copy_service.cc:161] P 655ebda1fb2c49c4af7087a760529835: Beginning new tablet copy session on tablet a1ebf29682024645978b3caa2d4988f3 from peer 60e8857abe7c41038af46ebb5a095a90 at {username='slave'} at 127.0.99.133:38419: session id = 60e8857abe7c41038af46ebb5a095a90-a1ebf29682024645978b3caa2d4988f3
I20240501 07:59:57.516165 15402 ts_tablet_manager.cc:1367] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608: Time spent bootstrapping tablet: real 0.062s user 0.024s sys 0.003s
I20240501 07:59:57.516695 15402 raft_consensus.cc:348] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:57.516898 15402 raft_consensus.cc:729] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca177138f2c44d2eafa1646b795f7608, State: Initialized, Role: FOLLOWER
I20240501 07:59:57.516902 11420 tablet_copy_source_session.cc:205] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:57.516916 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:57.517127 15402 consensus_queue.cc:260] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:57.517838 14918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1ebf29682024645978b3caa2d4988f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:57.518574 15402 ts_tablet_manager.cc:1398] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:57.519757 11420 tablet_copy_service.cc:342] P 655ebda1fb2c49c4af7087a760529835: Request end of tablet copy session ca177138f2c44d2eafa1646b795f7608-164fd265561e47128e7ff585b8cfab07 received from {username='slave'} at 127.0.99.136:50157
I20240501 07:59:57.519970 11420 tablet_copy_service.cc:433] P 655ebda1fb2c49c4af7087a760529835: ending tablet copy session ca177138f2c44d2eafa1646b795f7608-164fd265561e47128e7ff585b8cfab07 on tablet 164fd265561e47128e7ff585b8cfab07 with peer ca177138f2c44d2eafa1646b795f7608
I20240501 07:59:57.524379 14918 tablet_copy_client.cc:783] T a1ebf29682024645978b3caa2d4988f3 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:57.524747 14918 tablet_copy_client.cc:647] T a1ebf29682024645978b3caa2d4988f3 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:57.535274 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.536599 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.539306 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:57.546543 14958 ts_tablet_manager.cc:902] T c95c4f3ef08f44d1b69fb7b98426f5af P 60e8857abe7c41038af46ebb5a095a90: Initiating tablet copy from peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 07:59:57.547171 14958 tablet_copy_client.cc:308] T c95c4f3ef08f44d1b69fb7b98426f5af P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46085
I20240501 07:59:57.547838 11290 tablet_copy_service.cc:140] P e2dafc1866314669ad0de8fb7bf7fd05: Received BeginTabletCopySession request for tablet c95c4f3ef08f44d1b69fb7b98426f5af from peer 60e8857abe7c41038af46ebb5a095a90 ({username='slave'} at 127.0.99.133:60295)
I20240501 07:59:57.547952 11290 tablet_copy_service.cc:161] P e2dafc1866314669ad0de8fb7bf7fd05: Beginning new tablet copy session on tablet c95c4f3ef08f44d1b69fb7b98426f5af from peer 60e8857abe7c41038af46ebb5a095a90 at {username='slave'} at 127.0.99.133:60295: session id = 60e8857abe7c41038af46ebb5a095a90-c95c4f3ef08f44d1b69fb7b98426f5af
I20240501 07:59:57.548944 11290 tablet_copy_source_session.cc:205] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:57.550122 14958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95c4f3ef08f44d1b69fb7b98426f5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:57.553092 14958 tablet_copy_client.cc:783] T c95c4f3ef08f44d1b69fb7b98426f5af P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:57.553339 14958 tablet_copy_client.cc:647] T c95c4f3ef08f44d1b69fb7b98426f5af P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:57.554391 14918 tablet_copy_client.cc:523] T a1ebf29682024645978b3caa2d4988f3 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:57.555917 14918 tablet_bootstrap.cc:492] T a1ebf29682024645978b3caa2d4988f3 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap starting.
I20240501 07:59:57.557602 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.560710 13054 raft_consensus.cc:1203] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 07:59:57.563480 15402 ts_tablet_manager.cc:902] T 3f6e4ae97e884219ac7d0d94997d9cc9 P ca177138f2c44d2eafa1646b795f7608: Initiating tablet copy from peer 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:57.564019 15402 tablet_copy_client.cc:308] T 3f6e4ae97e884219ac7d0d94997d9cc9 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41575
I20240501 07:59:57.565101 11420 tablet_copy_service.cc:140] P 655ebda1fb2c49c4af7087a760529835: Received BeginTabletCopySession request for tablet 3f6e4ae97e884219ac7d0d94997d9cc9 from peer ca177138f2c44d2eafa1646b795f7608 ({username='slave'} at 127.0.99.136:50157)
I20240501 07:59:57.565228 11420 tablet_copy_service.cc:161] P 655ebda1fb2c49c4af7087a760529835: Beginning new tablet copy session on tablet 3f6e4ae97e884219ac7d0d94997d9cc9 from peer ca177138f2c44d2eafa1646b795f7608 at {username='slave'} at 127.0.99.136:50157: session id = ca177138f2c44d2eafa1646b795f7608-3f6e4ae97e884219ac7d0d94997d9cc9
I20240501 07:59:57.566155 11420 tablet_copy_source_session.cc:205] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:57.567173 15402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6e4ae97e884219ac7d0d94997d9cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:57.569404 15402 tablet_copy_client.cc:783] T 3f6e4ae97e884219ac7d0d94997d9cc9 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:57.569576 15402 tablet_copy_client.cc:647] T 3f6e4ae97e884219ac7d0d94997d9cc9 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:57.571275 15402 tablet_copy_client.cc:523] T 3f6e4ae97e884219ac7d0d94997d9cc9 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:57.572593 15402 tablet_bootstrap.cc:492] T 3f6e4ae97e884219ac7d0d94997d9cc9 P ca177138f2c44d2eafa1646b795f7608: Bootstrap starting.
I20240501 07:59:57.574786 14560 ts_tablet_manager.cc:902] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9: Initiating tablet copy from peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:57.575239 14560 tablet_copy_client.cc:308] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:44403
I20240501 07:59:57.575794 11683 tablet_copy_service.cc:140] P 603919bb02364c04b6c530245b8b02f2: Received BeginTabletCopySession request for tablet f61e70ba02384f81b9ae36b1c65465b9 from peer 53d1d2c0767b4f6195b69758a3dd6cf9 ({username='slave'} at 127.0.99.134:57705)
I20240501 07:59:57.575906 11683 tablet_copy_service.cc:161] P 603919bb02364c04b6c530245b8b02f2: Beginning new tablet copy session on tablet f61e70ba02384f81b9ae36b1c65465b9 from peer 53d1d2c0767b4f6195b69758a3dd6cf9 at {username='slave'} at 127.0.99.134:57705: session id = 53d1d2c0767b4f6195b69758a3dd6cf9-f61e70ba02384f81b9ae36b1c65465b9
I20240501 07:59:57.576802 11683 tablet_copy_source_session.cc:205] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:57.578702 14560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61e70ba02384f81b9ae36b1c65465b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:57.581509 14560 tablet_copy_client.cc:783] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:57.581699 14560 tablet_copy_client.cc:647] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:57.582969 14560 tablet_copy_client.cc:523] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:57.584870 14560 tablet_bootstrap.cc:492] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap starting.
I20240501 07:59:57.605770 15402 tablet_bootstrap.cc:492] T 3f6e4ae97e884219ac7d0d94997d9cc9 P ca177138f2c44d2eafa1646b795f7608: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:57.606178 15402 tablet_bootstrap.cc:492] T 3f6e4ae97e884219ac7d0d94997d9cc9 P ca177138f2c44d2eafa1646b795f7608: Bootstrap complete.
I20240501 07:59:57.606442 15402 ts_tablet_manager.cc:1367] T 3f6e4ae97e884219ac7d0d94997d9cc9 P ca177138f2c44d2eafa1646b795f7608: Time spent bootstrapping tablet: real 0.034s user 0.031s sys 0.000s
I20240501 07:59:57.607055 15402 raft_consensus.cc:348] T 3f6e4ae97e884219ac7d0d94997d9cc9 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:57.607533 15402 raft_consensus.cc:729] T 3f6e4ae97e884219ac7d0d94997d9cc9 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca177138f2c44d2eafa1646b795f7608, State: Initialized, Role: FOLLOWER
I20240501 07:59:57.607748 15402 consensus_queue.cc:260] T 3f6e4ae97e884219ac7d0d94997d9cc9 P ca177138f2c44d2eafa1646b795f7608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:57.608799 15402 ts_tablet_manager.cc:1398] T 3f6e4ae97e884219ac7d0d94997d9cc9 P ca177138f2c44d2eafa1646b795f7608: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:57.609397 11420 tablet_copy_service.cc:342] P 655ebda1fb2c49c4af7087a760529835: Request end of tablet copy session ca177138f2c44d2eafa1646b795f7608-3f6e4ae97e884219ac7d0d94997d9cc9 received from {username='slave'} at 127.0.99.136:50157
I20240501 07:59:57.609493 11420 tablet_copy_service.cc:433] P 655ebda1fb2c49c4af7087a760529835: ending tablet copy session ca177138f2c44d2eafa1646b795f7608-3f6e4ae97e884219ac7d0d94997d9cc9 on tablet 3f6e4ae97e884219ac7d0d94997d9cc9 with peer ca177138f2c44d2eafa1646b795f7608
I20240501 07:59:57.614816 14659 ts_tablet_manager.cc:902] T a1ebf29682024645978b3caa2d4988f3 P e5c03261d31c4d9a91077a17e1cd0336: Initiating tablet copy from peer 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:57.615298 14659 tablet_copy_client.cc:308] T a1ebf29682024645978b3caa2d4988f3 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41575
I20240501 07:59:57.615756 11420 tablet_copy_service.cc:140] P 655ebda1fb2c49c4af7087a760529835: Received BeginTabletCopySession request for tablet a1ebf29682024645978b3caa2d4988f3 from peer e5c03261d31c4d9a91077a17e1cd0336 ({username='slave'} at 127.0.99.135:48791)
I20240501 07:59:57.615877 11420 tablet_copy_service.cc:161] P 655ebda1fb2c49c4af7087a760529835: Beginning new tablet copy session on tablet a1ebf29682024645978b3caa2d4988f3 from peer e5c03261d31c4d9a91077a17e1cd0336 at {username='slave'} at 127.0.99.135:48791: session id = e5c03261d31c4d9a91077a17e1cd0336-a1ebf29682024645978b3caa2d4988f3
I20240501 07:59:57.616852 11420 tablet_copy_source_session.cc:205] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:57.618954 14659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1ebf29682024645978b3caa2d4988f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:57.621187 14659 tablet_copy_client.cc:783] T a1ebf29682024645978b3caa2d4988f3 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:57.621340 14659 tablet_copy_client.cc:647] T a1ebf29682024645978b3caa2d4988f3 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:57.622748 14659 tablet_copy_client.cc:523] T a1ebf29682024645978b3caa2d4988f3 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:57.624593 14659 tablet_bootstrap.cc:492] T a1ebf29682024645978b3caa2d4988f3 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap starting.
I20240501 07:59:57.625720 14560 tablet_bootstrap.cc:492] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:57.626005 14560 tablet_bootstrap.cc:492] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap complete.
I20240501 07:59:57.626178 14560 ts_tablet_manager.cc:1367] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent bootstrapping tablet: real 0.041s user 0.021s sys 0.005s
I20240501 07:59:57.627228 14560 raft_consensus.cc:348] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:57.627373 14560 raft_consensus.cc:729] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d1d2c0767b4f6195b69758a3dd6cf9, State: Initialized, Role: FOLLOWER
I20240501 07:59:57.627538 14560 consensus_queue.cc:260] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:57.628419 14560 ts_tablet_manager.cc:1398] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:57.628927 11683 tablet_copy_service.cc:342] P 603919bb02364c04b6c530245b8b02f2: Request end of tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-f61e70ba02384f81b9ae36b1c65465b9 received from {username='slave'} at 127.0.99.134:57705
I20240501 07:59:57.629014 11683 tablet_copy_service.cc:433] P 603919bb02364c04b6c530245b8b02f2: ending tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-f61e70ba02384f81b9ae36b1c65465b9 on tablet f61e70ba02384f81b9ae36b1c65465b9 with peer 53d1d2c0767b4f6195b69758a3dd6cf9
I20240501 07:59:57.630962 13185 raft_consensus.cc:1203] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:57.631558 14958 tablet_copy_client.cc:523] T c95c4f3ef08f44d1b69fb7b98426f5af P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:57.633072 14958 tablet_bootstrap.cc:492] T c95c4f3ef08f44d1b69fb7b98426f5af P 60e8857abe7c41038af46ebb5a095a90: Bootstrap starting.
I20240501 07:59:57.635320 15402 ts_tablet_manager.cc:902] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608: Initiating tablet copy from peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:57.635840 15402 tablet_copy_client.cc:308] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:44403
I20240501 07:59:57.641096 11401 consensus_queue.cc:237] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:57.642848 11269 raft_consensus.cc:1261] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:57.642956 11662 raft_consensus.cc:1261] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:57.643437 15127 consensus_queue.cc:1035] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:57.643756 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c47aaeb16c945ae84ece505aada2a7c. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:57.643927 15127 consensus_queue.cc:1035] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:57.650022 11683 tablet_copy_service.cc:140] P 603919bb02364c04b6c530245b8b02f2: Received BeginTabletCopySession request for tablet 53193305f868433c852755e8a8a81766 from peer ca177138f2c44d2eafa1646b795f7608 ({username='slave'} at 127.0.99.136:46561)
I20240501 07:59:57.650182 11683 tablet_copy_service.cc:161] P 603919bb02364c04b6c530245b8b02f2: Beginning new tablet copy session on tablet 53193305f868433c852755e8a8a81766 from peer ca177138f2c44d2eafa1646b795f7608 at {username='slave'} at 127.0.99.136:46561: session id = ca177138f2c44d2eafa1646b795f7608-53193305f868433c852755e8a8a81766
I20240501 07:59:57.651196 11683 tablet_copy_source_session.cc:205] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:57.653760 15402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53193305f868433c852755e8a8a81766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:57.656064 15402 tablet_copy_client.cc:783] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:57.656222 15402 tablet_copy_client.cc:647] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:57.659878 15127 raft_consensus.cc:2932] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.664319 11269 raft_consensus.cc:2932] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.664695 11126 catalog_manager.cc:5484] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.666313 11662 raft_consensus.cc:2932] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.673981 15402 tablet_copy_client.cc:523] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:57.675626 15402 tablet_bootstrap.cc:492] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608: Bootstrap starting.
I20240501 07:59:57.681545 11530 consensus_queue.cc:237] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:57.684105 11269 raft_consensus.cc:1261] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:57.684422 11401 raft_consensus.cc:1261] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:57.684722 14338 consensus_queue.cc:1035] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:57.685082 14338 consensus_queue.cc:1035] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:57.687052 13054 raft_consensus.cc:1203] T ed88ea311f474ab28cc6fbe6069258bf P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:57.689647 14918 tablet_bootstrap.cc:492] T a1ebf29682024645978b3caa2d4988f3 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:57.692888 14918 tablet_bootstrap.cc:492] T a1ebf29682024645978b3caa2d4988f3 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap complete.
I20240501 07:59:57.693228 14918 ts_tablet_manager.cc:1367] T a1ebf29682024645978b3caa2d4988f3 P 60e8857abe7c41038af46ebb5a095a90: Time spent bootstrapping tablet: real 0.137s user 0.028s sys 0.003s
I20240501 07:59:57.693773 14918 raft_consensus.cc:348] T a1ebf29682024645978b3caa2d4988f3 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:57.693903 14918 raft_consensus.cc:729] T a1ebf29682024645978b3caa2d4988f3 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e8857abe7c41038af46ebb5a095a90, State: Initialized, Role: FOLLOWER
I20240501 07:59:57.694072 14918 consensus_queue.cc:260] T a1ebf29682024645978b3caa2d4988f3 P 60e8857abe7c41038af46ebb5a095a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:57.692907 14958 tablet_bootstrap.cc:492] T c95c4f3ef08f44d1b69fb7b98426f5af P 60e8857abe7c41038af46ebb5a095a90: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:57.695173 14958 tablet_bootstrap.cc:492] T c95c4f3ef08f44d1b69fb7b98426f5af P 60e8857abe7c41038af46ebb5a095a90: Bootstrap complete.
I20240501 07:59:57.695587 14958 ts_tablet_manager.cc:1367] T c95c4f3ef08f44d1b69fb7b98426f5af P 60e8857abe7c41038af46ebb5a095a90: Time spent bootstrapping tablet: real 0.063s user 0.029s sys 0.000s
I20240501 07:59:57.697223 14958 raft_consensus.cc:348] T c95c4f3ef08f44d1b69fb7b98426f5af P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:57.708420 14659 tablet_bootstrap.cc:492] T a1ebf29682024645978b3caa2d4988f3 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:57.708736 14659 tablet_bootstrap.cc:492] T a1ebf29682024645978b3caa2d4988f3 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap complete.
I20240501 07:59:57.708940 14659 ts_tablet_manager.cc:1367] T a1ebf29682024645978b3caa2d4988f3 P e5c03261d31c4d9a91077a17e1cd0336: Time spent bootstrapping tablet: real 0.084s user 0.027s sys 0.000s
I20240501 07:59:57.709446 14659 raft_consensus.cc:348] T a1ebf29682024645978b3caa2d4988f3 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:57.709578 14659 raft_consensus.cc:729] T a1ebf29682024645978b3caa2d4988f3 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c03261d31c4d9a91077a17e1cd0336, State: Initialized, Role: FOLLOWER
I20240501 07:59:57.709712 14659 consensus_queue.cc:260] T a1ebf29682024645978b3caa2d4988f3 P e5c03261d31c4d9a91077a17e1cd0336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:57.709988 14659 ts_tablet_manager.cc:1398] T a1ebf29682024645978b3caa2d4988f3 P e5c03261d31c4d9a91077a17e1cd0336: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:57.710523 11420 tablet_copy_service.cc:342] P 655ebda1fb2c49c4af7087a760529835: Request end of tablet copy session e5c03261d31c4d9a91077a17e1cd0336-a1ebf29682024645978b3caa2d4988f3 received from {username='slave'} at 127.0.99.135:48791
I20240501 07:59:57.710613 11420 tablet_copy_service.cc:433] P 655ebda1fb2c49c4af7087a760529835: ending tablet copy session e5c03261d31c4d9a91077a17e1cd0336-a1ebf29682024645978b3caa2d4988f3 on tablet a1ebf29682024645978b3caa2d4988f3 with peer e5c03261d31c4d9a91077a17e1cd0336
I20240501 07:59:57.710749 14958 raft_consensus.cc:729] T c95c4f3ef08f44d1b69fb7b98426f5af P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e8857abe7c41038af46ebb5a095a90, State: Initialized, Role: FOLLOWER
I20240501 07:59:57.696517 14918 ts_tablet_manager.cc:1398] T a1ebf29682024645978b3caa2d4988f3 P 60e8857abe7c41038af46ebb5a095a90: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:59:57.711061 14958 consensus_queue.cc:260] T c95c4f3ef08f44d1b69fb7b98426f5af P 60e8857abe7c41038af46ebb5a095a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:57.712430 11420 tablet_copy_service.cc:342] P 655ebda1fb2c49c4af7087a760529835: Request end of tablet copy session 60e8857abe7c41038af46ebb5a095a90-a1ebf29682024645978b3caa2d4988f3 received from {username='slave'} at 127.0.99.133:38419
I20240501 07:59:57.712522 11420 tablet_copy_service.cc:433] P 655ebda1fb2c49c4af7087a760529835: ending tablet copy session 60e8857abe7c41038af46ebb5a095a90-a1ebf29682024645978b3caa2d4988f3 on tablet a1ebf29682024645978b3caa2d4988f3 with peer 60e8857abe7c41038af46ebb5a095a90
I20240501 07:59:57.712947 14958 ts_tablet_manager.cc:1398] T c95c4f3ef08f44d1b69fb7b98426f5af P 60e8857abe7c41038af46ebb5a095a90: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20240501 07:59:57.713481 11290 tablet_copy_service.cc:342] P e2dafc1866314669ad0de8fb7bf7fd05: Request end of tablet copy session 60e8857abe7c41038af46ebb5a095a90-c95c4f3ef08f44d1b69fb7b98426f5af received from {username='slave'} at 127.0.99.133:60295
I20240501 07:59:57.713562 11290 tablet_copy_service.cc:433] P e2dafc1866314669ad0de8fb7bf7fd05: ending tablet copy session 60e8857abe7c41038af46ebb5a095a90-c95c4f3ef08f44d1b69fb7b98426f5af on tablet c95c4f3ef08f44d1b69fb7b98426f5af with peer 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:57.716656 11468 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4707b952d174a9bb877ad8bd7a11268. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:57.718283 14350 raft_consensus.cc:2932] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.720731 11401 raft_consensus.cc:2932] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.722015 11269 raft_consensus.cc:2932] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.723182 11120 catalog_manager.cc:5484] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:57.754926 15402 tablet_bootstrap.cc:492] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:57.756409 15402 tablet_bootstrap.cc:492] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608: Bootstrap complete.
I20240501 07:59:57.756717 15402 ts_tablet_manager.cc:1367] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608: Time spent bootstrapping tablet: real 0.081s user 0.025s sys 0.004s
I20240501 07:59:57.758219 15402 raft_consensus.cc:348] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:57.760989 15402 raft_consensus.cc:729] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca177138f2c44d2eafa1646b795f7608, State: Initialized, Role: FOLLOWER
I20240501 07:59:57.761243 15402 consensus_queue.cc:260] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:57.761603 15402 ts_tablet_manager.cc:1398] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:59:57.762197 11683 tablet_copy_service.cc:342] P 603919bb02364c04b6c530245b8b02f2: Request end of tablet copy session ca177138f2c44d2eafa1646b795f7608-53193305f868433c852755e8a8a81766 received from {username='slave'} at 127.0.99.136:46561
I20240501 07:59:57.762282 11683 tablet_copy_service.cc:433] P 603919bb02364c04b6c530245b8b02f2: ending tablet copy session ca177138f2c44d2eafa1646b795f7608-53193305f868433c852755e8a8a81766 on tablet 53193305f868433c852755e8a8a81766 with peer ca177138f2c44d2eafa1646b795f7608
I20240501 07:59:57.768447 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:57.771534 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.781821 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.783129 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:57.800454 13185 raft_consensus.cc:1203] T b4d15e19eed54aefb6b96276fad98324 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:57.806927 11530 consensus_queue.cc:237] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:57.809083 11269 raft_consensus.cc:1261] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:57.809579 15081 consensus_queue.cc:1035] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:57.810117 11401 raft_consensus.cc:1261] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:57.810617 11465 consensus_peers.cc:485] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771): Couldn't send request to peer 53d1d2c0767b4f6195b69758a3dd6cf9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17bb1ff829bc47a6a0bd6030517652aa. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:57.811180 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:57.811581 15081 consensus_queue.cc:1035] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:57.824700 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.824726 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:57.841601 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.842730 14659 ts_tablet_manager.cc:902] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336: Initiating tablet copy from peer 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:57.843286 14659 tablet_copy_client.cc:308] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41575
I20240501 07:59:57.843883 11420 tablet_copy_service.cc:140] P 655ebda1fb2c49c4af7087a760529835: Received BeginTabletCopySession request for tablet 7c23e3bd7ec34876956046e5d32e4d82 from peer e5c03261d31c4d9a91077a17e1cd0336 ({username='slave'} at 127.0.99.135:48791)
I20240501 07:59:57.844018 11420 tablet_copy_service.cc:161] P 655ebda1fb2c49c4af7087a760529835: Beginning new tablet copy session on tablet 7c23e3bd7ec34876956046e5d32e4d82 from peer e5c03261d31c4d9a91077a17e1cd0336 at {username='slave'} at 127.0.99.135:48791: session id = e5c03261d31c4d9a91077a17e1cd0336-7c23e3bd7ec34876956046e5d32e4d82
I20240501 07:59:57.845060 11420 tablet_copy_source_session.cc:205] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:57.846674 14659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c23e3bd7ec34876956046e5d32e4d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:57.849252 14659 tablet_copy_client.cc:783] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:57.849453 14659 tablet_copy_client.cc:647] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:57.851387 15081 raft_consensus.cc:2932] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:57.853662 11269 raft_consensus.cc:2932] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:57.854039 11400 raft_consensus.cc:2932] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:57.854446 11128 catalog_manager.cc:5484] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index -1 to 4, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:57.873039 14659 tablet_copy_client.cc:523] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:57.874962 14659 tablet_bootstrap.cc:492] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap starting.
I20240501 07:59:57.875432 14560 ts_tablet_manager.cc:902] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9: Initiating tablet copy from peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:57.875929 14560 tablet_copy_client.cc:308] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37957
I20240501 07:59:57.876559 11552 tablet_copy_service.cc:140] P 22ec12774a4047a8b859743dc73f8ff6: Received BeginTabletCopySession request for tablet 48b0ff0769dc426190f038811155c741 from peer 53d1d2c0767b4f6195b69758a3dd6cf9 ({username='slave'} at 127.0.99.134:40967)
I20240501 07:59:57.876691 11552 tablet_copy_service.cc:161] P 22ec12774a4047a8b859743dc73f8ff6: Beginning new tablet copy session on tablet 48b0ff0769dc426190f038811155c741 from peer 53d1d2c0767b4f6195b69758a3dd6cf9 at {username='slave'} at 127.0.99.134:40967: session id = 53d1d2c0767b4f6195b69758a3dd6cf9-48b0ff0769dc426190f038811155c741
I20240501 07:59:57.877730 11552 tablet_copy_source_session.cc:205] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:57.879530 14560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b0ff0769dc426190f038811155c741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:57.882156 14560 tablet_copy_client.cc:783] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:57.882366 14560 tablet_copy_client.cc:647] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:57.884997 13317 raft_consensus.cc:1203] T 6d214be86534458091f54a386d7d3334 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 07:59:57.885553 13449 raft_consensus.cc:1203] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 07:59:57.889331 14560 tablet_copy_client.cc:523] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:57.891094 14560 tablet_bootstrap.cc:492] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap starting.
I20240501 07:59:57.911823 11400 consensus_queue.cc:237] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:57.913631 11269 raft_consensus.cc:1261] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:57.914721 11662 raft_consensus.cc:1261] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 07:59:57.917260 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c47aaeb16c945ae84ece505aada2a7c. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:57.920189 14430 consensus_queue.cc:1035] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:57.922057 14430 consensus_queue.cc:1035] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:57.922153 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.926321 14659 tablet_bootstrap.cc:492] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:57.926668 14659 tablet_bootstrap.cc:492] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap complete.
I20240501 07:59:57.926863 14659 ts_tablet_manager.cc:1367] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336: Time spent bootstrapping tablet: real 0.052s user 0.017s sys 0.007s
W20240501 07:59:57.927315 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c47aaeb16c945ae84ece505aada2a7c. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:57.927377 14659 raft_consensus.cc:348] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:57.927491 14659 raft_consensus.cc:729] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c03261d31c4d9a91077a17e1cd0336, State: Initialized, Role: FOLLOWER
I20240501 07:59:57.927650 14659 consensus_queue.cc:260] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:57.927981 14659 ts_tablet_manager.cc:1398] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:57.928246 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:57.930807 14430 raft_consensus.cc:2932] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:57.931123 11420 tablet_copy_service.cc:342] P 655ebda1fb2c49c4af7087a760529835: Request end of tablet copy session e5c03261d31c4d9a91077a17e1cd0336-7c23e3bd7ec34876956046e5d32e4d82 received from {username='slave'} at 127.0.99.135:48791
I20240501 07:59:57.931210 11420 tablet_copy_service.cc:433] P 655ebda1fb2c49c4af7087a760529835: ending tablet copy session e5c03261d31c4d9a91077a17e1cd0336-7c23e3bd7ec34876956046e5d32e4d82 on tablet 7c23e3bd7ec34876956046e5d32e4d82 with peer e5c03261d31c4d9a91077a17e1cd0336
I20240501 07:59:57.932179 11663 raft_consensus.cc:2932] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:57.935667 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:57.936393 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.936758 11270 raft_consensus.cc:2932] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:57.937561 11128 catalog_manager.cc:5484] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 4 to 5, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:57.939967 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.959523 11662 consensus_queue.cc:237] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:57.960955 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:57.962080 11400 raft_consensus.cc:1261] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:57.962930 13507 consensus_queue.cc:1035] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:57.963456 11598 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b21b6e2023e842739a7c0cc946e7c2fd. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:57.964578 11270 raft_consensus.cc:1261] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:57.965502 14660 consensus_queue.cc:1035] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:57.966326 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:57.977258 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:57.988655 13449 raft_consensus.cc:1203] T 3f6e4ae97e884219ac7d0d94997d9cc9 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 07:59:58.005643 13507 raft_consensus.cc:2932] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:58.010658 11269 raft_consensus.cc:2932] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:58.013936 11126 catalog_manager.cc:5484] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:58.017297 11401 raft_consensus.cc:2932] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:58.035485 14560 tablet_bootstrap.cc:492] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:58.036198 14560 tablet_bootstrap.cc:492] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap complete.
I20240501 07:59:58.036461 14560 ts_tablet_manager.cc:1367] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent bootstrapping tablet: real 0.145s user 0.024s sys 0.004s
I20240501 07:59:58.037061 14560 raft_consensus.cc:348] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:58.037288 14560 raft_consensus.cc:729] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d1d2c0767b4f6195b69758a3dd6cf9, State: Initialized, Role: FOLLOWER
I20240501 07:59:58.037504 14560 consensus_queue.cc:260] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:58.037889 14560 ts_tablet_manager.cc:1398] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:58.039630 11552 tablet_copy_service.cc:342] P 22ec12774a4047a8b859743dc73f8ff6: Request end of tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-48b0ff0769dc426190f038811155c741 received from {username='slave'} at 127.0.99.134:40967
I20240501 07:59:58.039745 11552 tablet_copy_service.cc:433] P 22ec12774a4047a8b859743dc73f8ff6: ending tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-48b0ff0769dc426190f038811155c741 on tablet 48b0ff0769dc426190f038811155c741 with peer 53d1d2c0767b4f6195b69758a3dd6cf9
I20240501 07:59:58.048640 11530 consensus_queue.cc:237] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
W20240501 07:59:58.050789 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4707b952d174a9bb877ad8bd7a11268. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:58.052037 11269 raft_consensus.cc:1261] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:58.052623 14338 consensus_queue.cc:1035] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 07:59:58.053256 11468 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4707b952d174a9bb877ad8bd7a11268. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:58.056023 11401 raft_consensus.cc:1261] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:58.056555 14338 consensus_queue.cc:1035] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:58.058753 14338 raft_consensus.cc:2932] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:58.059849 11401 raft_consensus.cc:2932] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:58.062595 11269 raft_consensus.cc:2932] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:58.063926 11126 catalog_manager.cc:5484] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 4 to 5, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:58.079280 13054 raft_consensus.cc:1203] T c95c4f3ef08f44d1b69fb7b98426f5af P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:58.087360 13054 raft_consensus.cc:1203] T a1ebf29682024645978b3caa2d4988f3 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 07:59:58.119174 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:58.125175 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:58.125485 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:58.126040 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.127105 13185 raft_consensus.cc:1203] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 07:59:58.129813 13449 raft_consensus.cc:1203] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:58.132383 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:58.149580 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.151149 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.159123 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.180109 13317 raft_consensus.cc:1203] T a1ebf29682024645978b3caa2d4988f3 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 07:59:58.222657 11269 consensus_queue.cc:237] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:58.224372 11532 raft_consensus.cc:1261] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:58.224562 11662 raft_consensus.cc:1261] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:58.224819 14240 consensus_queue.cc:1035] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:58.225945 14240 consensus_queue.cc:1035] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:58.226282 11206 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a24be92aec7544089c7818f2802b9c85. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:58.232802 15771 raft_consensus.cc:2932] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:58.235381 11662 raft_consensus.cc:2932] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:58.237527 11532 raft_consensus.cc:2932] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:58.241120 11126 catalog_manager.cc:5484] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index -1 to 4, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:58.252858 11269 consensus_queue.cc:237] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
W20240501 07:59:58.254886 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a266f266c3d4fd49fa8b325f1045d61. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:58.255152 11401 raft_consensus.cc:1261] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:58.255626 14241 consensus_queue.cc:1035] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:58.256834 11662 raft_consensus.cc:1261] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:58.257529 15882 consensus_queue.cc:1035] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:58.263294 14241 raft_consensus.cc:2932] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:58.266753 11662 raft_consensus.cc:2932] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:58.266753 11401 raft_consensus.cc:2932] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:58.267114 11126 catalog_manager.cc:5484] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index -1 to 4, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 07:59:58.281719 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.292037 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:58.294823 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:58.296679 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:58.296949 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:58.302955 14659 ts_tablet_manager.cc:902] T 5c47aaeb16c945ae84ece505aada2a7c P e5c03261d31c4d9a91077a17e1cd0336: Initiating tablet copy from peer 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:58.303452 14659 tablet_copy_client.cc:308] T 5c47aaeb16c945ae84ece505aada2a7c P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41575
I20240501 07:59:58.303999 11420 tablet_copy_service.cc:140] P 655ebda1fb2c49c4af7087a760529835: Received BeginTabletCopySession request for tablet 5c47aaeb16c945ae84ece505aada2a7c from peer e5c03261d31c4d9a91077a17e1cd0336 ({username='slave'} at 127.0.99.135:48791)
I20240501 07:59:58.304095 11420 tablet_copy_service.cc:161] P 655ebda1fb2c49c4af7087a760529835: Beginning new tablet copy session on tablet 5c47aaeb16c945ae84ece505aada2a7c from peer e5c03261d31c4d9a91077a17e1cd0336 at {username='slave'} at 127.0.99.135:48791: session id = e5c03261d31c4d9a91077a17e1cd0336-5c47aaeb16c945ae84ece505aada2a7c
I20240501 07:59:58.304798 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.305431 11420 tablet_copy_source_session.cc:205] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:58.306735 14659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c47aaeb16c945ae84ece505aada2a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:58.306887 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.309669 14659 tablet_copy_client.cc:783] T 5c47aaeb16c945ae84ece505aada2a7c P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:58.309859 14659 tablet_copy_client.cc:647] T 5c47aaeb16c945ae84ece505aada2a7c P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:58.318589 14560 ts_tablet_manager.cc:902] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9: Initiating tablet copy from peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:58.319078 14560 tablet_copy_client.cc:308] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37957
I20240501 07:59:58.326534 11552 tablet_copy_service.cc:140] P 22ec12774a4047a8b859743dc73f8ff6: Received BeginTabletCopySession request for tablet 17bb1ff829bc47a6a0bd6030517652aa from peer 53d1d2c0767b4f6195b69758a3dd6cf9 ({username='slave'} at 127.0.99.134:40967)
I20240501 07:59:58.326707 11552 tablet_copy_service.cc:161] P 22ec12774a4047a8b859743dc73f8ff6: Beginning new tablet copy session on tablet 17bb1ff829bc47a6a0bd6030517652aa from peer 53d1d2c0767b4f6195b69758a3dd6cf9 at {username='slave'} at 127.0.99.134:40967: session id = 53d1d2c0767b4f6195b69758a3dd6cf9-17bb1ff829bc47a6a0bd6030517652aa
I20240501 07:59:58.327728 11552 tablet_copy_source_session.cc:205] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:58.332085 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.345746 14560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17bb1ff829bc47a6a0bd6030517652aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:58.348932 14560 tablet_copy_client.cc:783] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:58.349130 14560 tablet_copy_client.cc:647] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:58.349368 14659 tablet_copy_client.cc:523] T 5c47aaeb16c945ae84ece505aada2a7c P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:58.351017 14659 tablet_bootstrap.cc:492] T 5c47aaeb16c945ae84ece505aada2a7c P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap starting.
I20240501 07:59:58.374715 15941 ts_tablet_manager.cc:902] T 5c47aaeb16c945ae84ece505aada2a7c P ca177138f2c44d2eafa1646b795f7608: Initiating tablet copy from peer 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 07:59:58.375007 15953 ts_tablet_manager.cc:902] T b21b6e2023e842739a7c0cc946e7c2fd P 60e8857abe7c41038af46ebb5a095a90: Initiating tablet copy from peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 07:59:58.375588 15953 tablet_copy_client.cc:308] T b21b6e2023e842739a7c0cc946e7c2fd P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:44403
I20240501 07:59:58.376255 11683 tablet_copy_service.cc:140] P 603919bb02364c04b6c530245b8b02f2: Received BeginTabletCopySession request for tablet b21b6e2023e842739a7c0cc946e7c2fd from peer 60e8857abe7c41038af46ebb5a095a90 ({username='slave'} at 127.0.99.133:51087)
I20240501 07:59:58.376365 11683 tablet_copy_service.cc:161] P 603919bb02364c04b6c530245b8b02f2: Beginning new tablet copy session on tablet b21b6e2023e842739a7c0cc946e7c2fd from peer 60e8857abe7c41038af46ebb5a095a90 at {username='slave'} at 127.0.99.133:51087: session id = 60e8857abe7c41038af46ebb5a095a90-b21b6e2023e842739a7c0cc946e7c2fd
I20240501 07:59:58.376550 15941 tablet_copy_client.cc:308] T 5c47aaeb16c945ae84ece505aada2a7c P ca177138f2c44d2eafa1646b795f7608: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41575
I20240501 07:59:58.377300 11683 tablet_copy_source_session.cc:205] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:58.378733 11420 tablet_copy_service.cc:140] P 655ebda1fb2c49c4af7087a760529835: Received BeginTabletCopySession request for tablet 5c47aaeb16c945ae84ece505aada2a7c from peer ca177138f2c44d2eafa1646b795f7608 ({username='slave'} at 127.0.99.136:50157)
I20240501 07:59:58.378856 11420 tablet_copy_service.cc:161] P 655ebda1fb2c49c4af7087a760529835: Beginning new tablet copy session on tablet 5c47aaeb16c945ae84ece505aada2a7c from peer ca177138f2c44d2eafa1646b795f7608 at {username='slave'} at 127.0.99.136:50157: session id = ca177138f2c44d2eafa1646b795f7608-5c47aaeb16c945ae84ece505aada2a7c
I20240501 07:59:58.379799 11420 tablet_copy_source_session.cc:205] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:58.381347 13186 raft_consensus.cc:1203] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:58.388883 15953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21b6e2023e842739a7c0cc946e7c2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:58.389379 15941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c47aaeb16c945ae84ece505aada2a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:58.394659 14560 tablet_copy_client.cc:523] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:58.396456 14560 tablet_bootstrap.cc:492] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap starting.
I20240501 07:59:58.398912 15941 tablet_copy_client.cc:783] T 5c47aaeb16c945ae84ece505aada2a7c P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:58.399477 15941 tablet_copy_client.cc:647] T 5c47aaeb16c945ae84ece505aada2a7c P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:58.399605 15953 tablet_copy_client.cc:783] T b21b6e2023e842739a7c0cc946e7c2fd P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:58.400089 15953 tablet_copy_client.cc:647] T b21b6e2023e842739a7c0cc946e7c2fd P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:58.403076 15941 tablet_copy_client.cc:523] T 5c47aaeb16c945ae84ece505aada2a7c P ca177138f2c44d2eafa1646b795f7608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:58.403151 14659 tablet_bootstrap.cc:492] T 5c47aaeb16c945ae84ece505aada2a7c P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:58.403412 14659 tablet_bootstrap.cc:492] T 5c47aaeb16c945ae84ece505aada2a7c P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap complete.
I20240501 07:59:58.403606 14659 ts_tablet_manager.cc:1367] T 5c47aaeb16c945ae84ece505aada2a7c P e5c03261d31c4d9a91077a17e1cd0336: Time spent bootstrapping tablet: real 0.053s user 0.022s sys 0.003s
I20240501 07:59:58.404160 14659 raft_consensus.cc:348] T 5c47aaeb16c945ae84ece505aada2a7c P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:58.404296 14659 raft_consensus.cc:729] T 5c47aaeb16c945ae84ece505aada2a7c P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c03261d31c4d9a91077a17e1cd0336, State: Initialized, Role: FOLLOWER
I20240501 07:59:58.404459 14659 consensus_queue.cc:260] T 5c47aaeb16c945ae84ece505aada2a7c P e5c03261d31c4d9a91077a17e1cd0336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:58.404759 14659 ts_tablet_manager.cc:1398] T 5c47aaeb16c945ae84ece505aada2a7c P e5c03261d31c4d9a91077a17e1cd0336: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:58.404991 15953 tablet_copy_client.cc:523] T b21b6e2023e842739a7c0cc946e7c2fd P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:58.405233 11420 tablet_copy_service.cc:342] P 655ebda1fb2c49c4af7087a760529835: Request end of tablet copy session e5c03261d31c4d9a91077a17e1cd0336-5c47aaeb16c945ae84ece505aada2a7c received from {username='slave'} at 127.0.99.135:48791
I20240501 07:59:58.405318 11420 tablet_copy_service.cc:433] P 655ebda1fb2c49c4af7087a760529835: ending tablet copy session e5c03261d31c4d9a91077a17e1cd0336-5c47aaeb16c945ae84ece505aada2a7c on tablet 5c47aaeb16c945ae84ece505aada2a7c with peer e5c03261d31c4d9a91077a17e1cd0336
I20240501 07:59:58.408423 15953 tablet_bootstrap.cc:492] T b21b6e2023e842739a7c0cc946e7c2fd P 60e8857abe7c41038af46ebb5a095a90: Bootstrap starting.
I20240501 07:59:58.408921 15941 tablet_bootstrap.cc:492] T 5c47aaeb16c945ae84ece505aada2a7c P ca177138f2c44d2eafa1646b795f7608: Bootstrap starting.
I20240501 07:59:58.411374 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.412067 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.414395 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:58.428121 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:58.438920 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:58.444160 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:58.446645 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.455914 13317 raft_consensus.cc:1203] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:58.490907 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.531075 15988 ts_tablet_manager.cc:902] T b4707b952d174a9bb877ad8bd7a11268 P 60e8857abe7c41038af46ebb5a095a90: Initiating tablet copy from peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:58.531700 15988 tablet_copy_client.cc:308] T b4707b952d174a9bb877ad8bd7a11268 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37957
I20240501 07:59:58.532400 11552 tablet_copy_service.cc:140] P 22ec12774a4047a8b859743dc73f8ff6: Received BeginTabletCopySession request for tablet b4707b952d174a9bb877ad8bd7a11268 from peer 60e8857abe7c41038af46ebb5a095a90 ({username='slave'} at 127.0.99.133:59721)
I20240501 07:59:58.532529 11552 tablet_copy_service.cc:161] P 22ec12774a4047a8b859743dc73f8ff6: Beginning new tablet copy session on tablet b4707b952d174a9bb877ad8bd7a11268 from peer 60e8857abe7c41038af46ebb5a095a90 at {username='slave'} at 127.0.99.133:59721: session id = 60e8857abe7c41038af46ebb5a095a90-b4707b952d174a9bb877ad8bd7a11268
I20240501 07:59:58.533581 11552 tablet_copy_source_session.cc:205] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:58.535034 15988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4707b952d174a9bb877ad8bd7a11268. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:58.538623 15988 tablet_copy_client.cc:783] T b4707b952d174a9bb877ad8bd7a11268 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:58.538913 15988 tablet_copy_client.cc:647] T b4707b952d174a9bb877ad8bd7a11268 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:58.542258 15988 tablet_copy_client.cc:523] T b4707b952d174a9bb877ad8bd7a11268 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:58.543988 15988 tablet_bootstrap.cc:492] T b4707b952d174a9bb877ad8bd7a11268 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap starting.
I20240501 07:59:58.556424 11270 consensus_queue.cc:237] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
W20240501 07:59:58.559022 11206 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a24be92aec7544089c7818f2802b9c85. This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:58.559356 11203 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771): Couldn't send request to peer 53d1d2c0767b4f6195b69758a3dd6cf9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a24be92aec7544089c7818f2802b9c85. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:58.560036 11532 raft_consensus.cc:1261] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:58.560950 11662 raft_consensus.cc:1261] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:58.561225 14240 consensus_queue.cc:1035] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:58.561414 15882 consensus_queue.cc:1035] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:58.566632 15941 tablet_bootstrap.cc:492] T 5c47aaeb16c945ae84ece505aada2a7c P ca177138f2c44d2eafa1646b795f7608: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:58.567031 15941 tablet_bootstrap.cc:492] T 5c47aaeb16c945ae84ece505aada2a7c P ca177138f2c44d2eafa1646b795f7608: Bootstrap complete.
I20240501 07:59:58.568810 14240 raft_consensus.cc:2932] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:58.569411 11532 raft_consensus.cc:2932] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:58.571368 15941 ts_tablet_manager.cc:1367] T 5c47aaeb16c945ae84ece505aada2a7c P ca177138f2c44d2eafa1646b795f7608: Time spent bootstrapping tablet: real 0.162s user 0.027s sys 0.001s
I20240501 07:59:58.571605 11128 catalog_manager.cc:5484] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index 4 to 5, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:58.571914 15941 raft_consensus.cc:348] T 5c47aaeb16c945ae84ece505aada2a7c P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:58.572026 15941 raft_consensus.cc:729] T 5c47aaeb16c945ae84ece505aada2a7c P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca177138f2c44d2eafa1646b795f7608, State: Initialized, Role: FOLLOWER
I20240501 07:59:58.572188 15941 consensus_queue.cc:260] T 5c47aaeb16c945ae84ece505aada2a7c P ca177138f2c44d2eafa1646b795f7608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:58.573100 15941 ts_tablet_manager.cc:1398] T 5c47aaeb16c945ae84ece505aada2a7c P ca177138f2c44d2eafa1646b795f7608: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:58.574012 11420 tablet_copy_service.cc:342] P 655ebda1fb2c49c4af7087a760529835: Request end of tablet copy session ca177138f2c44d2eafa1646b795f7608-5c47aaeb16c945ae84ece505aada2a7c received from {username='slave'} at 127.0.99.136:50157
I20240501 07:59:58.574086 11420 tablet_copy_service.cc:433] P 655ebda1fb2c49c4af7087a760529835: ending tablet copy session ca177138f2c44d2eafa1646b795f7608-5c47aaeb16c945ae84ece505aada2a7c on tablet 5c47aaeb16c945ae84ece505aada2a7c with peer ca177138f2c44d2eafa1646b795f7608
I20240501 07:59:58.577899 15941 ts_tablet_manager.cc:902] T b4707b952d174a9bb877ad8bd7a11268 P ca177138f2c44d2eafa1646b795f7608: Initiating tablet copy from peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 07:59:58.579447 15941 tablet_copy_client.cc:308] T b4707b952d174a9bb877ad8bd7a11268 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37957
I20240501 07:59:58.591420 11662 raft_consensus.cc:2932] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:58.591753 15953 tablet_bootstrap.cc:492] T b21b6e2023e842739a7c0cc946e7c2fd P 60e8857abe7c41038af46ebb5a095a90: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:58.592073 15953 tablet_bootstrap.cc:492] T b21b6e2023e842739a7c0cc946e7c2fd P 60e8857abe7c41038af46ebb5a095a90: Bootstrap complete.
I20240501 07:59:58.592260 15953 ts_tablet_manager.cc:1367] T b21b6e2023e842739a7c0cc946e7c2fd P 60e8857abe7c41038af46ebb5a095a90: Time spent bootstrapping tablet: real 0.184s user 0.025s sys 0.004s
I20240501 07:59:58.595089 15953 raft_consensus.cc:348] T b21b6e2023e842739a7c0cc946e7c2fd P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:58.595247 15953 raft_consensus.cc:729] T b21b6e2023e842739a7c0cc946e7c2fd P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e8857abe7c41038af46ebb5a095a90, State: Initialized, Role: FOLLOWER
I20240501 07:59:58.595389 15953 consensus_queue.cc:260] T b21b6e2023e842739a7c0cc946e7c2fd P 60e8857abe7c41038af46ebb5a095a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:58.596835 14560 tablet_bootstrap.cc:492] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:58.598137 11552 tablet_copy_service.cc:140] P 22ec12774a4047a8b859743dc73f8ff6: Received BeginTabletCopySession request for tablet b4707b952d174a9bb877ad8bd7a11268 from peer ca177138f2c44d2eafa1646b795f7608 ({username='slave'} at 127.0.99.136:39189)
I20240501 07:59:58.598251 11552 tablet_copy_service.cc:161] P 22ec12774a4047a8b859743dc73f8ff6: Beginning new tablet copy session on tablet b4707b952d174a9bb877ad8bd7a11268 from peer ca177138f2c44d2eafa1646b795f7608 at {username='slave'} at 127.0.99.136:39189: session id = ca177138f2c44d2eafa1646b795f7608-b4707b952d174a9bb877ad8bd7a11268
I20240501 07:59:58.598376 14560 tablet_bootstrap.cc:492] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap complete.
I20240501 07:59:58.599864 11552 tablet_copy_source_session.cc:205] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:58.600924 15941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4707b952d174a9bb877ad8bd7a11268. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:58.601320 14560 ts_tablet_manager.cc:1367] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent bootstrapping tablet: real 0.205s user 0.029s sys 0.000s
I20240501 07:59:58.602316 14560 raft_consensus.cc:348] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:58.602459 14560 raft_consensus.cc:729] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d1d2c0767b4f6195b69758a3dd6cf9, State: Initialized, Role: FOLLOWER
I20240501 07:59:58.602595 14560 consensus_queue.cc:260] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:58.603076 14560 ts_tablet_manager.cc:1398] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:58.603531 11552 tablet_copy_service.cc:342] P 22ec12774a4047a8b859743dc73f8ff6: Request end of tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-17bb1ff829bc47a6a0bd6030517652aa received from {username='slave'} at 127.0.99.134:40967
I20240501 07:59:58.603610 11552 tablet_copy_service.cc:433] P 22ec12774a4047a8b859743dc73f8ff6: ending tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-17bb1ff829bc47a6a0bd6030517652aa on tablet 17bb1ff829bc47a6a0bd6030517652aa with peer 53d1d2c0767b4f6195b69758a3dd6cf9
I20240501 07:59:58.595762 15953 ts_tablet_manager.cc:1398] T b21b6e2023e842739a7c0cc946e7c2fd P 60e8857abe7c41038af46ebb5a095a90: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:58.605654 11683 tablet_copy_service.cc:342] P 603919bb02364c04b6c530245b8b02f2: Request end of tablet copy session 60e8857abe7c41038af46ebb5a095a90-b21b6e2023e842739a7c0cc946e7c2fd received from {username='slave'} at 127.0.99.133:51087
I20240501 07:59:58.605746 11683 tablet_copy_service.cc:433] P 603919bb02364c04b6c530245b8b02f2: ending tablet copy session 60e8857abe7c41038af46ebb5a095a90-b21b6e2023e842739a7c0cc946e7c2fd on tablet b21b6e2023e842739a7c0cc946e7c2fd with peer 60e8857abe7c41038af46ebb5a095a90
I20240501 07:59:58.606065 15941 tablet_copy_client.cc:783] T b4707b952d174a9bb877ad8bd7a11268 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:58.606360 15941 tablet_copy_client.cc:647] T b4707b952d174a9bb877ad8bd7a11268 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:58.609824 15941 tablet_copy_client.cc:523] T b4707b952d174a9bb877ad8bd7a11268 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:58.612591 15941 tablet_bootstrap.cc:492] T b4707b952d174a9bb877ad8bd7a11268 P ca177138f2c44d2eafa1646b795f7608: Bootstrap starting.
I20240501 07:59:58.645566 15941 tablet_bootstrap.cc:492] T b4707b952d174a9bb877ad8bd7a11268 P ca177138f2c44d2eafa1646b795f7608: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:58.648770 15941 tablet_bootstrap.cc:492] T b4707b952d174a9bb877ad8bd7a11268 P ca177138f2c44d2eafa1646b795f7608: Bootstrap complete.
I20240501 07:59:58.649471 15941 ts_tablet_manager.cc:1367] T b4707b952d174a9bb877ad8bd7a11268 P ca177138f2c44d2eafa1646b795f7608: Time spent bootstrapping tablet: real 0.037s user 0.027s sys 0.000s
I20240501 07:59:58.650067 15941 raft_consensus.cc:348] T b4707b952d174a9bb877ad8bd7a11268 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:58.650267 15941 raft_consensus.cc:729] T b4707b952d174a9bb877ad8bd7a11268 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca177138f2c44d2eafa1646b795f7608, State: Initialized, Role: FOLLOWER
I20240501 07:59:58.650476 15941 consensus_queue.cc:260] T b4707b952d174a9bb877ad8bd7a11268 P ca177138f2c44d2eafa1646b795f7608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:58.651333 15941 ts_tablet_manager.cc:1398] T b4707b952d174a9bb877ad8bd7a11268 P ca177138f2c44d2eafa1646b795f7608: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:58.651855 11552 tablet_copy_service.cc:342] P 22ec12774a4047a8b859743dc73f8ff6: Request end of tablet copy session ca177138f2c44d2eafa1646b795f7608-b4707b952d174a9bb877ad8bd7a11268 received from {username='slave'} at 127.0.99.136:39189
I20240501 07:59:58.651938 11552 tablet_copy_service.cc:433] P 22ec12774a4047a8b859743dc73f8ff6: ending tablet copy session ca177138f2c44d2eafa1646b795f7608-b4707b952d174a9bb877ad8bd7a11268 on tablet b4707b952d174a9bb877ad8bd7a11268 with peer ca177138f2c44d2eafa1646b795f7608
I20240501 07:59:58.703148 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:58.707662 15988 tablet_bootstrap.cc:492] T b4707b952d174a9bb877ad8bd7a11268 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:58.708015 15988 tablet_bootstrap.cc:492] T b4707b952d174a9bb877ad8bd7a11268 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap complete.
I20240501 07:59:58.708201 15988 ts_tablet_manager.cc:1367] T b4707b952d174a9bb877ad8bd7a11268 P 60e8857abe7c41038af46ebb5a095a90: Time spent bootstrapping tablet: real 0.164s user 0.026s sys 0.000s
I20240501 07:59:58.708684 15988 raft_consensus.cc:348] T b4707b952d174a9bb877ad8bd7a11268 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:58.708817 15988 raft_consensus.cc:729] T b4707b952d174a9bb877ad8bd7a11268 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e8857abe7c41038af46ebb5a095a90, State: Initialized, Role: FOLLOWER
I20240501 07:59:58.708977 15988 consensus_queue.cc:260] T b4707b952d174a9bb877ad8bd7a11268 P 60e8857abe7c41038af46ebb5a095a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:58.709259 15988 ts_tablet_manager.cc:1398] T b4707b952d174a9bb877ad8bd7a11268 P 60e8857abe7c41038af46ebb5a095a90: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:58.710734 11552 tablet_copy_service.cc:342] P 22ec12774a4047a8b859743dc73f8ff6: Request end of tablet copy session 60e8857abe7c41038af46ebb5a095a90-b4707b952d174a9bb877ad8bd7a11268 received from {username='slave'} at 127.0.99.133:59721
I20240501 07:59:58.710824 11552 tablet_copy_service.cc:433] P 22ec12774a4047a8b859743dc73f8ff6: ending tablet copy session 60e8857abe7c41038af46ebb5a095a90-b4707b952d174a9bb877ad8bd7a11268 on tablet b4707b952d174a9bb877ad8bd7a11268 with peer 60e8857abe7c41038af46ebb5a095a90
I20240501 07:59:58.719435 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.720345 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.732606 15988 ts_tablet_manager.cc:902] T 7a266f266c3d4fd49fa8b325f1045d61 P 60e8857abe7c41038af46ebb5a095a90: Initiating tablet copy from peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 07:59:58.732618 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:58.735745 15988 tablet_copy_client.cc:308] T 7a266f266c3d4fd49fa8b325f1045d61 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46085
I20240501 07:59:58.740983 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:58.749089 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.749843 11290 tablet_copy_service.cc:140] P e2dafc1866314669ad0de8fb7bf7fd05: Received BeginTabletCopySession request for tablet 7a266f266c3d4fd49fa8b325f1045d61 from peer 60e8857abe7c41038af46ebb5a095a90 ({username='slave'} at 127.0.99.133:60295)
I20240501 07:59:58.749979 11290 tablet_copy_service.cc:161] P e2dafc1866314669ad0de8fb7bf7fd05: Beginning new tablet copy session on tablet 7a266f266c3d4fd49fa8b325f1045d61 from peer 60e8857abe7c41038af46ebb5a095a90 at {username='slave'} at 127.0.99.133:60295: session id = 60e8857abe7c41038af46ebb5a095a90-7a266f266c3d4fd49fa8b325f1045d61
I20240501 07:59:58.750972 11290 tablet_copy_source_session.cc:205] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:58.752111 15988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a266f266c3d4fd49fa8b325f1045d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:58.755604 15988 tablet_copy_client.cc:783] T 7a266f266c3d4fd49fa8b325f1045d61 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:58.755883 15988 tablet_copy_client.cc:647] T 7a266f266c3d4fd49fa8b325f1045d61 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:58.758886 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:58.763672 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.765070 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:58.766906 13317 raft_consensus.cc:1203] T 5c47aaeb16c945ae84ece505aada2a7c P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 07:59:58.768997 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:58.770411 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.775449 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.782936 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:58.792977 13186 raft_consensus.cc:1203] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:58.799763 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.813166 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:58.821265 13054 raft_consensus.cc:1203] T b21b6e2023e842739a7c0cc946e7c2fd P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:58.823218 13448 raft_consensus.cc:1203] T 5c47aaeb16c945ae84ece505aada2a7c P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 07:59:58.838827 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.884368 15988 tablet_copy_client.cc:523] T 7a266f266c3d4fd49fa8b325f1045d61 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:58.886025 15988 tablet_bootstrap.cc:492] T 7a266f266c3d4fd49fa8b325f1045d61 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap starting.
I20240501 07:59:58.900503 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:58.915460 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:58.918308 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.918648 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.934652 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:58.937774 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.950115 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:58.950695 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:58.966322 15988 tablet_bootstrap.cc:492] T 7a266f266c3d4fd49fa8b325f1045d61 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:58.966668 15988 tablet_bootstrap.cc:492] T 7a266f266c3d4fd49fa8b325f1045d61 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap complete.
I20240501 07:59:58.966873 15988 ts_tablet_manager.cc:1367] T 7a266f266c3d4fd49fa8b325f1045d61 P 60e8857abe7c41038af46ebb5a095a90: Time spent bootstrapping tablet: real 0.081s user 0.030s sys 0.000s
I20240501 07:59:58.967399 15988 raft_consensus.cc:348] T 7a266f266c3d4fd49fa8b325f1045d61 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:58.967533 15988 raft_consensus.cc:729] T 7a266f266c3d4fd49fa8b325f1045d61 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e8857abe7c41038af46ebb5a095a90, State: Initialized, Role: FOLLOWER
I20240501 07:59:58.967670 15988 consensus_queue.cc:260] T 7a266f266c3d4fd49fa8b325f1045d61 P 60e8857abe7c41038af46ebb5a095a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 07:59:58.967972 15988 ts_tablet_manager.cc:1398] T 7a266f266c3d4fd49fa8b325f1045d61 P 60e8857abe7c41038af46ebb5a095a90: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:58.968447 11290 tablet_copy_service.cc:342] P e2dafc1866314669ad0de8fb7bf7fd05: Request end of tablet copy session 60e8857abe7c41038af46ebb5a095a90-7a266f266c3d4fd49fa8b325f1045d61 received from {username='slave'} at 127.0.99.133:60295
I20240501 07:59:58.968549 11290 tablet_copy_service.cc:433] P e2dafc1866314669ad0de8fb7bf7fd05: ending tablet copy session 60e8857abe7c41038af46ebb5a095a90-7a266f266c3d4fd49fa8b325f1045d61 on tablet 7a266f266c3d4fd49fa8b325f1045d61 with peer 60e8857abe7c41038af46ebb5a095a90
I20240501 07:59:58.969641 13054 raft_consensus.cc:1203] T b4707b952d174a9bb877ad8bd7a11268 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 07:59:59.010084 13448 raft_consensus.cc:1203] T b4707b952d174a9bb877ad8bd7a11268 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 07:59:59.027194 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:59.027532 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:59.038476 11512 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:59.042371 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.048511 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.054742 15941 ts_tablet_manager.cc:902] T a24be92aec7544089c7818f2802b9c85 P ca177138f2c44d2eafa1646b795f7608: Initiating tablet copy from peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 07:59:59.055254 15941 tablet_copy_client.cc:308] T a24be92aec7544089c7818f2802b9c85 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46085
I20240501 07:59:59.055989 11290 tablet_copy_service.cc:140] P e2dafc1866314669ad0de8fb7bf7fd05: Received BeginTabletCopySession request for tablet a24be92aec7544089c7818f2802b9c85 from peer ca177138f2c44d2eafa1646b795f7608 ({username='slave'} at 127.0.99.136:55185)
I20240501 07:59:59.056092 11290 tablet_copy_service.cc:161] P e2dafc1866314669ad0de8fb7bf7fd05: Beginning new tablet copy session on tablet a24be92aec7544089c7818f2802b9c85 from peer ca177138f2c44d2eafa1646b795f7608 at {username='slave'} at 127.0.99.136:55185: session id = ca177138f2c44d2eafa1646b795f7608-a24be92aec7544089c7818f2802b9c85
I20240501 07:59:59.056245 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.057045 11290 tablet_copy_source_session.cc:205] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:59.058458 14560 ts_tablet_manager.cc:902] T a24be92aec7544089c7818f2802b9c85 P 53d1d2c0767b4f6195b69758a3dd6cf9: Initiating tablet copy from peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 07:59:59.059650 15941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a24be92aec7544089c7818f2802b9c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:59.062018 15941 tablet_copy_client.cc:783] T a24be92aec7544089c7818f2802b9c85 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:59.062594 15941 tablet_copy_client.cc:647] T a24be92aec7544089c7818f2802b9c85 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:59.066020 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.066712 14560 tablet_copy_client.cc:308] T a24be92aec7544089c7818f2802b9c85 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46085
I20240501 07:59:59.067266 11290 tablet_copy_service.cc:140] P e2dafc1866314669ad0de8fb7bf7fd05: Received BeginTabletCopySession request for tablet a24be92aec7544089c7818f2802b9c85 from peer 53d1d2c0767b4f6195b69758a3dd6cf9 ({username='slave'} at 127.0.99.134:37773)
I20240501 07:59:59.067378 11290 tablet_copy_service.cc:161] P e2dafc1866314669ad0de8fb7bf7fd05: Beginning new tablet copy session on tablet a24be92aec7544089c7818f2802b9c85 from peer 53d1d2c0767b4f6195b69758a3dd6cf9 at {username='slave'} at 127.0.99.134:37773: session id = 53d1d2c0767b4f6195b69758a3dd6cf9-a24be92aec7544089c7818f2802b9c85
I20240501 07:59:59.068409 11290 tablet_copy_source_session.cc:205] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:59.069578 14560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a24be92aec7544089c7818f2802b9c85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:59.075527 14560 tablet_copy_client.cc:783] T a24be92aec7544089c7818f2802b9c85 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:59.076540 14560 tablet_copy_client.cc:647] T a24be92aec7544089c7818f2802b9c85 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:59.076769 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.085888 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:59.099912 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:59.103547 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.111191 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:59.113399 14560 tablet_copy_client.cc:523] T a24be92aec7544089c7818f2802b9c85 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:59.117219 14560 tablet_bootstrap.cc:492] T a24be92aec7544089c7818f2802b9c85 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap starting.
I20240501 07:59:59.120365 13054 raft_consensus.cc:1203] T 7a266f266c3d4fd49fa8b325f1045d61 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:59.121057 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:59.122105 15941 tablet_copy_client.cc:523] T a24be92aec7544089c7818f2802b9c85 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:59.124202 15941 tablet_bootstrap.cc:492] T a24be92aec7544089c7818f2802b9c85 P ca177138f2c44d2eafa1646b795f7608: Bootstrap starting.
I20240501 07:59:59.133803 11380 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:59.143378 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.163838 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.186405 11401 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "90f11b5693d7456d8a604e6c8726e69a"
dest_uuid: "655ebda1fb2c49c4af7087a760529835"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48504
I20240501 07:59:59.186638 11401 raft_consensus.cc:595] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:59.199678 15941 tablet_bootstrap.cc:492] T a24be92aec7544089c7818f2802b9c85 P ca177138f2c44d2eafa1646b795f7608: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:59.200125 15941 tablet_bootstrap.cc:492] T a24be92aec7544089c7818f2802b9c85 P ca177138f2c44d2eafa1646b795f7608: Bootstrap complete.
I20240501 07:59:59.200346 15941 ts_tablet_manager.cc:1367] T a24be92aec7544089c7818f2802b9c85 P ca177138f2c44d2eafa1646b795f7608: Time spent bootstrapping tablet: real 0.076s user 0.025s sys 0.000s
I20240501 07:59:59.200882 15941 raft_consensus.cc:348] T a24be92aec7544089c7818f2802b9c85 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:59.201375 15941 raft_consensus.cc:729] T a24be92aec7544089c7818f2802b9c85 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca177138f2c44d2eafa1646b795f7608, State: Initialized, Role: FOLLOWER
I20240501 07:59:59.201524 15941 consensus_queue.cc:260] T a24be92aec7544089c7818f2802b9c85 P ca177138f2c44d2eafa1646b795f7608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:59.202033 15941 ts_tablet_manager.cc:1398] T a24be92aec7544089c7818f2802b9c85 P ca177138f2c44d2eafa1646b795f7608: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:59.202616 11290 tablet_copy_service.cc:342] P e2dafc1866314669ad0de8fb7bf7fd05: Request end of tablet copy session ca177138f2c44d2eafa1646b795f7608-a24be92aec7544089c7818f2802b9c85 received from {username='slave'} at 127.0.99.136:55185
I20240501 07:59:59.202702 11290 tablet_copy_service.cc:433] P e2dafc1866314669ad0de8fb7bf7fd05: ending tablet copy session ca177138f2c44d2eafa1646b795f7608-a24be92aec7544089c7818f2802b9c85 on tablet a24be92aec7544089c7818f2802b9c85 with peer ca177138f2c44d2eafa1646b795f7608
I20240501 07:59:59.229202 14560 tablet_bootstrap.cc:492] T a24be92aec7544089c7818f2802b9c85 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:59.229620 14560 tablet_bootstrap.cc:492] T a24be92aec7544089c7818f2802b9c85 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap complete.
I20240501 07:59:59.229885 14560 ts_tablet_manager.cc:1367] T a24be92aec7544089c7818f2802b9c85 P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent bootstrapping tablet: real 0.113s user 0.031s sys 0.000s
I20240501 07:59:59.230448 14560 raft_consensus.cc:348] T a24be92aec7544089c7818f2802b9c85 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:59.230592 14560 raft_consensus.cc:729] T a24be92aec7544089c7818f2802b9c85 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d1d2c0767b4f6195b69758a3dd6cf9, State: Initialized, Role: FOLLOWER
I20240501 07:59:59.230755 14560 consensus_queue.cc:260] T a24be92aec7544089c7818f2802b9c85 P 53d1d2c0767b4f6195b69758a3dd6cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:59.231062 14560 ts_tablet_manager.cc:1398] T a24be92aec7544089c7818f2802b9c85 P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:59.231652 11290 tablet_copy_service.cc:342] P e2dafc1866314669ad0de8fb7bf7fd05: Request end of tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-a24be92aec7544089c7818f2802b9c85 received from {username='slave'} at 127.0.99.134:37773
I20240501 07:59:59.231730 11290 tablet_copy_service.cc:433] P e2dafc1866314669ad0de8fb7bf7fd05: ending tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-a24be92aec7544089c7818f2802b9c85 on tablet a24be92aec7544089c7818f2802b9c85 with peer 53d1d2c0767b4f6195b69758a3dd6cf9
I20240501 07:59:59.248514 11532 consensus_queue.cc:237] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:59.250743 13186 raft_consensus.cc:1261] T 93602521de174997bda6702d25aafce1 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:59.251149 11662 raft_consensus.cc:1261] T 93602521de174997bda6702d25aafce1 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:59.253165 15582 consensus_queue.cc:1035] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:59.253610 15582 consensus_queue.cc:1035] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:59.258060 15081 raft_consensus.cc:2932] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:59.264046 11662 raft_consensus.cc:2932] T 93602521de174997bda6702d25aafce1 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:59.266624 13186 raft_consensus.cc:2932] T 93602521de174997bda6702d25aafce1 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:59.267884 11126 catalog_manager.cc:5484] T 93602521de174997bda6702d25aafce1 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 4 to 5, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:59.283612 11400 consensus_queue.cc:237] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:59.286346 11380 tablet_service.cc:1514] Processing DeleteTablet for tablet 93602521de174997bda6702d25aafce1 with delete_type TABLET_DATA_TOMBSTONED (TS 655ebda1fb2c49c4af7087a760529835 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:43354
I20240501 07:59:59.290050 16388 tablet_replica.cc:330] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835: stopping tablet replica
I20240501 07:59:59.290275 13054 raft_consensus.cc:1261] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:59.290503 16388 raft_consensus.cc:2229] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:59.290809 16388 raft_consensus.cc:2258] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:59.291170 15692 consensus_queue.cc:1035] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:59.292433 11662 raft_consensus.cc:1261] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:59.292476 16388 ts_tablet_manager.cc:1875] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:59.293674 16387 consensus_queue.cc:1035] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:59.294080 13448 raft_consensus.cc:1261] T 3f6e4ae97e884219ac7d0d94997d9cc9 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:59.296334 16388 ts_tablet_manager.cc:1888] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:59:59.296578 16388 log.cc:1196] T 93602521de174997bda6702d25aafce1 P 655ebda1fb2c49c4af7087a760529835: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/93602521de174997bda6702d25aafce1
I20240501 07:59:59.297557 11112 catalog_manager.cc:4830] TS 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575): tablet 93602521de174997bda6702d25aafce1 (table rebalance_test_table_1 [id=98fb02bdb5f54b1699749e3c8d588a5e]) successfully deleted
I20240501 07:59:59.299798 15127 consensus_queue.cc:1035] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:59.300606 16387 raft_consensus.cc:979] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835: : Instructing follower 22ec12774a4047a8b859743dc73f8ff6 to start an election
I20240501 07:59:59.300755 16387 raft_consensus.cc:1067] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Signalling peer 22ec12774a4047a8b859743dc73f8ff6 to start an election
I20240501 07:59:59.301645 11532 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90f11b5693d7456d8a604e6c8726e69a"
dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
from {username='slave'} at 127.0.99.130:44689
I20240501 07:59:59.301834 11532 raft_consensus.cc:482] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:59.301910 11532 raft_consensus.cc:3037] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:59.303108 11532 raft_consensus.cc:504] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:59.303582 11532 leader_election.cc:288] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 2 election: Requested vote from peers 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575), ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503)
I20240501 07:59:59.304957 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f11b5693d7456d8a604e6c8726e69a" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ca177138f2c44d2eafa1646b795f7608"
I20240501 07:59:59.305157 13449 raft_consensus.cc:3037] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:59.308558 11663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f11b5693d7456d8a604e6c8726e69a" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 07:59:59.308718 11663 raft_consensus.cc:3037] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:59.309782 11663 raft_consensus.cc:2454] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 2.
I20240501 07:59:59.314255 11400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f11b5693d7456d8a604e6c8726e69a" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 07:59:59.314555 11400 raft_consensus.cc:3032] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:59.314615 11400 raft_consensus.cc:729] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Running, Role: LEADER
I20240501 07:59:59.314824 11400 consensus_queue.cc:260] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:59.315106 11400 raft_consensus.cc:3037] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:59.318316 15692 raft_consensus.cc:2932] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:59.319631 11662 raft_consensus.cc:2932] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:59.321875 11120 catalog_manager.cc:5484] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:59.323156 13054 raft_consensus.cc:2932] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:59.324072 13449 raft_consensus.cc:2454] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 2.
I20240501 07:59:59.324234 13446 raft_consensus.cc:1226] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Rejecting Update request from peer 655ebda1fb2c49c4af7087a760529835 for earlier term 1. Current term is 2. Ops: []
I20240501 07:59:59.325225 11468 leader_election.cc:302] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 603919bb02364c04b6c530245b8b02f2, ca177138f2c44d2eafa1646b795f7608; no voters:
I20240501 07:59:59.326599 11512 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6e4ae97e884219ac7d0d94997d9cc9 with delete_type TABLET_DATA_TOMBSTONED (TS 22ec12774a4047a8b859743dc73f8ff6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52810
I20240501 07:59:59.327075 11400 raft_consensus.cc:2454] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 2.
I20240501 07:59:59.327080 16360 raft_consensus.cc:2781] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:59.327183 16360 raft_consensus.cc:686] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Becoming Leader. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Running, Role: LEADER
I20240501 07:59:59.327344 16417 tablet_replica.cc:330] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6: stopping tablet replica
I20240501 07:59:59.327694 16417 raft_consensus.cc:2229] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:59.327912 16417 raft_consensus.cc:2258] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:59.329361 16417 ts_tablet_manager.cc:1875] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:59.327366 16360 consensus_queue.cc:237] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:59.331871 11127 catalog_manager.cc:5484] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: term changed from 1 to 2, leader changed from 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) to 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:59.332135 16417 ts_tablet_manager.cc:1888] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:59.332228 16417 log.cc:1196] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 22ec12774a4047a8b859743dc73f8ff6: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/3f6e4ae97e884219ac7d0d94997d9cc9
I20240501 07:59:59.333474 11114 catalog_manager.cc:4830] TS 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957): tablet 3f6e4ae97e884219ac7d0d94997d9cc9 (table rebalance_test_table_9 [id=1deda0b4866b43c9ba7c69dae5d65e4a]) successfully deleted
I20240501 07:59:59.343166 13448 raft_consensus.cc:2932] T 3f6e4ae97e884219ac7d0d94997d9cc9 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 07:59:59.396852 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.436760 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.438113 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.440330 11512 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:59.440438 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.440567 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:59.445847 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.446594 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:59.450608 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.455941 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.457448 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:59:59.471541 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:59.489110 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.501317 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:59.509210 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:59:59.516881 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:59.519834 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:59.529084 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.536296 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.562608 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.581076 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.581991 13186 raft_consensus.cc:1203] T a24be92aec7544089c7818f2802b9c85 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 07:59:59.596105 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:59.617133 13448 raft_consensus.cc:1203] T a24be92aec7544089c7818f2802b9c85 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 07:59:59.622718 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:59.623458 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:59:59.647079 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:59.658877 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.659328 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.669216 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.695879 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.709535 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.714637 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:59.719574 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:59.720569 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:59.727450 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:59:59.736936 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.738809 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.742035 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:59.742758 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:59.755964 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:59.797067 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:59:59.849865 11532 consensus_queue.cc:237] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:59.851940 11269 raft_consensus.cc:1261] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:59.851950 13316 raft_consensus.cc:1261] T 6d214be86534458091f54a386d7d3334 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:59.852428 13054 raft_consensus.cc:1261] T 6d214be86534458091f54a386d7d3334 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:59.852465 16360 consensus_queue.cc:1035] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:59.852792 16637 consensus_queue.cc:1035] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:59.853085 16360 consensus_queue.cc:1035] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:59.854120 11400 raft_consensus.cc:1261] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:59.854686 16360 consensus_queue.cc:1035] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:59.858937 16638 raft_consensus.cc:2932] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:59.861347 13316 raft_consensus.cc:2932] T 6d214be86534458091f54a386d7d3334 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:59.861959 11128 catalog_manager.cc:5484] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 5 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:59.865032 11532 consensus_queue.cc:237] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:59.864936 13054 raft_consensus.cc:2932] T 6d214be86534458091f54a386d7d3334 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:59.867060 13186 raft_consensus.cc:1261] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:59.867476 11662 raft_consensus.cc:1261] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:59.868081 16360 consensus_queue.cc:1035] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:59.868790 11269 raft_consensus.cc:2932] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 07:59:59.869446 16360 consensus_queue.cc:1035] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:59.871867 11642 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d214be86534458091f54a386d7d3334 with delete_type TABLET_DATA_TOMBSTONED (TS 603919bb02364c04b6c530245b8b02f2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53514
I20240501 07:59:59.874912 16360 raft_consensus.cc:2932] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:59.877528 16670 tablet_replica.cc:330] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2: stopping tablet replica
I20240501 07:59:59.877954 16670 raft_consensus.cc:2229] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:59.878276 16670 raft_consensus.cc:2258] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:59.878338 11662 consensus_queue.cc:237] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 07:59:59.879312 11663 raft_consensus.cc:2932] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:59.879494 16670 ts_tablet_manager.cc:1875] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:59.881428 13185 raft_consensus.cc:1261] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:59.881731 16670 ts_tablet_manager.cc:1888] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:59.881842 16670 log.cc:1196] T 6d214be86534458091f54a386d7d3334 P 603919bb02364c04b6c530245b8b02f2: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/6d214be86534458091f54a386d7d3334
I20240501 07:59:59.882030 16296 consensus_queue.cc:1035] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:59.882572 11112 catalog_manager.cc:4830] TS 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): tablet 6d214be86534458091f54a386d7d3334 (table rebalance_test_table_7 [id=36e20074d8dc4402bd7d9b2039213af5]) successfully deleted
I20240501 07:59:59.883066 11532 raft_consensus.cc:1261] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:59.883443 16296 consensus_queue.cc:1035] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:59.884487 11122 catalog_manager.cc:5484] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 4 to 5, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:59.886615 16296 raft_consensus.cc:2932] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:59.888092 13448 raft_consensus.cc:1261] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:59.888499 16637 consensus_queue.cc:1035] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:59.888687 11532 raft_consensus.cc:2932] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:59.889837 11380 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b0ff0769dc426190f038811155c741 with delete_type TABLET_DATA_TOMBSTONED (TS 655ebda1fb2c49c4af7087a760529835 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:43354
I20240501 07:59:59.893013 11663 raft_consensus.cc:1261] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:59.893926 11128 catalog_manager.cc:5484] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 reported cstate change: config changed from index 4 to 5, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:59.894001 16638 consensus_queue.cc:1035] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:59.895850 16693 tablet_replica.cc:330] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835: stopping tablet replica
I20240501 07:59:59.896070 16693 raft_consensus.cc:2229] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:59.896584 16693 raft_consensus.cc:2258] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:59.897215 16693 ts_tablet_manager.cc:1875] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:59.897882 11248 tablet_service.cc:1514] Processing DeleteTablet for tablet f80db2e37bbc4716ad3a5200c61ee97b with delete_type TABLET_DATA_TOMBSTONED (TS e2dafc1866314669ad0de8fb7bf7fd05 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42432
I20240501 07:59:59.902654 13186 raft_consensus.cc:2932] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:59.907445 16693 ts_tablet_manager.cc:1888] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:59:59.907636 16693 log.cc:1196] T 48b0ff0769dc426190f038811155c741 P 655ebda1fb2c49c4af7087a760529835: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/48b0ff0769dc426190f038811155c741
I20240501 07:59:59.907481 13185 raft_consensus.cc:2932] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 07:59:59.908291 11112 catalog_manager.cc:4830] TS 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575): tablet 48b0ff0769dc426190f038811155c741 (table rebalance_test_table_3 [id=6447707eb34f45d7a8d3704cd09d7786]) successfully deleted
I20240501 07:59:59.917873 16701 tablet_replica.cc:330] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05: stopping tablet replica
I20240501 07:59:59.919684 16701 raft_consensus.cc:2229] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:59.920760 16701 raft_consensus.cc:2258] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:59.921955 16701 ts_tablet_manager.cc:1875] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:59.924746 16701 ts_tablet_manager.cc:1888] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 07:59:59.926347 16701 log.cc:1196] T f80db2e37bbc4716ad3a5200c61ee97b P e2dafc1866314669ad0de8fb7bf7fd05: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f80db2e37bbc4716ad3a5200c61ee97b
I20240501 07:59:59.928822 11113 catalog_manager.cc:4830] TS e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085): tablet f80db2e37bbc4716ad3a5200c61ee97b (table rebalance_test_table_4 [id=384b2b245c6b4931a4fc02c6b5d647b2]) successfully deleted
I20240501 07:59:59.986789 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:00.003185 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.008713 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.018007 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:00.030128 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:00.050671 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:00.057736 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.059584 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.097640 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.105224 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.112161 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:00.119138 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:00.124981 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.126955 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:00.128180 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.132928 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.134652 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:00.137557 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:00.158629 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:00.162546 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.164192 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:00.187510 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:00.187981 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:00.210232 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.214948 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.226600 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.233922 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:00.241446 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.276209 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:00.279422 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.280417 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.313158 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.318264 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.326864 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:00.330752 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:00.340504 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:00.342357 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.342592 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:00.352916 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.358073 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:00.418548 11400 consensus_queue.cc:237] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:00.420531 11663 raft_consensus.cc:1261] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:00.420567 13316 raft_consensus.cc:1261] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:00.421198 15127 consensus_queue.cc:1035] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:00.421558 15127 consensus_queue.cc:1035] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:00.422719 11662 consensus_queue.cc:237] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:00.424108 15127 raft_consensus.cc:2932] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:00.424531 11662 raft_consensus.cc:2932] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:00.425801 13054 raft_consensus.cc:1261] T ed88ea311f474ab28cc6fbe6069258bf P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:00.426189 11532 raft_consensus.cc:1261] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:00.428337 16576 consensus_queue.cc:1035] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:00.428520 16678 consensus_queue.cc:1035] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:00.428866 11126 catalog_manager.cc:5484] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 4 to 5, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:00.434962 16947 raft_consensus.cc:2932] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:00.435561 11532 raft_consensus.cc:2932] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:00.438915 11119 catalog_manager.cc:5484] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 4 to 5, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:00.443212 13054 raft_consensus.cc:2932] T ed88ea311f474ab28cc6fbe6069258bf P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:00.446171 11510 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c23e3bd7ec34876956046e5d32e4d82 with delete_type TABLET_DATA_TOMBSTONED (TS 22ec12774a4047a8b859743dc73f8ff6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52810
I20240501 08:00:00.447206 11248 tablet_service.cc:1514] Processing DeleteTablet for tablet ed88ea311f474ab28cc6fbe6069258bf with delete_type TABLET_DATA_TOMBSTONED (TS e2dafc1866314669ad0de8fb7bf7fd05 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42432
I20240501 08:00:00.448915 16986 tablet_replica.cc:330] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6: stopping tablet replica
I20240501 08:00:00.449143 16986 raft_consensus.cc:2229] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:00.449487 13316 raft_consensus.cc:2932] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:00.450649 16986 raft_consensus.cc:2258] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:00.451279 16986 ts_tablet_manager.cc:1875] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:00.452270 16988 tablet_replica.cc:330] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: stopping tablet replica
I20240501 08:00:00.452443 16988 raft_consensus.cc:2229] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:00.452622 16988 raft_consensus.cc:2258] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:00.453194 16988 ts_tablet_manager.cc:1875] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:00.455641 16988 ts_tablet_manager.cc:1888] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:00:00.455801 16988 log.cc:1196] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ed88ea311f474ab28cc6fbe6069258bf
I20240501 08:00:00.456382 11113 catalog_manager.cc:4830] TS e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085): tablet ed88ea311f474ab28cc6fbe6069258bf (table rebalance_test_table_8 [id=df5f9864bbf8499dbaa392bcb85701c0]) successfully deleted
I20240501 08:00:00.458030 16986 ts_tablet_manager.cc:1888] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:00:00.458184 16986 log.cc:1196] T 7c23e3bd7ec34876956046e5d32e4d82 P 22ec12774a4047a8b859743dc73f8ff6: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/7c23e3bd7ec34876956046e5d32e4d82
I20240501 08:00:00.459004 11114 catalog_manager.cc:4830] TS 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957): tablet 7c23e3bd7ec34876956046e5d32e4d82 (table rebalance_test_table_9 [id=1deda0b4866b43c9ba7c69dae5d65e4a]) successfully deleted
I20240501 08:00:00.467021 11269 consensus_queue.cc:237] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:00.467506 11270 consensus_queue.cc:237] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:00.468753 11662 raft_consensus.cc:1261] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:00.468894 11532 raft_consensus.cc:1261] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:00.469115 16396 consensus_queue.cc:1035] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:00.469883 16396 consensus_queue.cc:1035] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:00.473088 16396 raft_consensus.cc:2932] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:00.473555 11662 raft_consensus.cc:2932] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:00.475646 13316 raft_consensus.cc:1261] T 18b146cc4e164e1d9f9f827217c9b3ab P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:00.476825 13448 raft_consensus.cc:1261] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:00.477538 16396 consensus_queue.cc:1035] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:00.477525 17008 consensus_queue.cc:1035] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:00.478180 11119 catalog_manager.cc:5484] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index 4 to 5, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:00.479125 13448 raft_consensus.cc:2932] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:00.479216 11532 raft_consensus.cc:2932] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:00.480213 17008 raft_consensus.cc:2932] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:00.483678 11510 tablet_service.cc:1514] Processing DeleteTablet for tablet 867ba08a15b44d4c9e6fa92b0d518d1d with delete_type TABLET_DATA_TOMBSTONED (TS 22ec12774a4047a8b859743dc73f8ff6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52810
I20240501 08:00:00.483968 16986 tablet_replica.cc:330] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6: stopping tablet replica
I20240501 08:00:00.484117 16986 raft_consensus.cc:2229] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:00.484256 16986 raft_consensus.cc:2258] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:00.484856 16986 ts_tablet_manager.cc:1875] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:00.488317 11124 catalog_manager.cc:5484] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 4 to 5, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:00.497092 11380 tablet_service.cc:1514] Processing DeleteTablet for tablet 18b146cc4e164e1d9f9f827217c9b3ab with delete_type TABLET_DATA_TOMBSTONED (TS 655ebda1fb2c49c4af7087a760529835 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:43354
I20240501 08:00:00.497531 17032 tablet_replica.cc:330] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835: stopping tablet replica
I20240501 08:00:00.497673 17032 raft_consensus.cc:2229] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:00.497864 17032 raft_consensus.cc:2258] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:00.498724 17032 ts_tablet_manager.cc:1875] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:00.498800 16986 ts_tablet_manager.cc:1888] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:00:00.498951 16986 log.cc:1196] T 867ba08a15b44d4c9e6fa92b0d518d1d P 22ec12774a4047a8b859743dc73f8ff6: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/867ba08a15b44d4c9e6fa92b0d518d1d
I20240501 08:00:00.499572 11114 catalog_manager.cc:4830] TS 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957): tablet 867ba08a15b44d4c9e6fa92b0d518d1d (table rebalance_test_table_3 [id=6447707eb34f45d7a8d3704cd09d7786]) successfully deleted
I20240501 08:00:00.501111 17032 ts_tablet_manager.cc:1888] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:00:00.501271 17032 log.cc:1196] T 18b146cc4e164e1d9f9f827217c9b3ab P 655ebda1fb2c49c4af7087a760529835: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/18b146cc4e164e1d9f9f827217c9b3ab
I20240501 08:00:00.501901 11112 catalog_manager.cc:4830] TS 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575): tablet 18b146cc4e164e1d9f9f827217c9b3ab (table rebalance_test_table_9 [id=1deda0b4866b43c9ba7c69dae5d65e4a]) successfully deleted
I20240501 08:00:00.505246 13316 raft_consensus.cc:2932] T 18b146cc4e164e1d9f9f827217c9b3ab P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:00.592180 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.604787 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:00.607342 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.622280 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:00.628844 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.641829 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:00.650655 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.654927 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:00.664377 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.665930 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:00.671841 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:00.672523 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.685359 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:00.685822 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.687804 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.717020 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:00.721886 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:00.728401 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:00.739238 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.745033 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.776232 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:00.776525 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:00.796155 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.835673 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.867925 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.873708 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:00.882889 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:00.884665 11401 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b9431cf5853c4148ac38855b1c083d4f"
dest_uuid: "655ebda1fb2c49c4af7087a760529835"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51988
I20240501 08:00:00.885125 11401 raft_consensus.cc:595] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:00.889560 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.889732 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:00.891072 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:00.897109 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.903136 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.904769 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:00.915400 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.917423 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.922919 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:00.947108 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:00.984028 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:00.985232 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.014134 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:01.022975 11401 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b9431cf5853c4148ac38855b1c083d4f"
dest_uuid: "655ebda1fb2c49c4af7087a760529835"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52002
I20240501 08:00:01.023135 11401 raft_consensus.cc:595] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:01.038885 11401 consensus_queue.cc:237] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:01.040786 13316 raft_consensus.cc:1261] T 5c47aaeb16c945ae84ece505aada2a7c P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:01.040905 11270 raft_consensus.cc:1261] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:01.041280 15127 consensus_queue.cc:1035] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:01.041442 13448 raft_consensus.cc:1261] T 5c47aaeb16c945ae84ece505aada2a7c P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:01.042043 17027 consensus_queue.cc:1035] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:01.042349 17027 consensus_queue.cc:1035] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:01.048326 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:01.052178 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:01.059640 15127 raft_consensus.cc:2932] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:01.064563 13448 raft_consensus.cc:2932] T 5c47aaeb16c945ae84ece505aada2a7c P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:01.064757 11121 catalog_manager.cc:5484] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 5 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:01.074870 11270 raft_consensus.cc:2932] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:01.076457 13316 raft_consensus.cc:2932] T 5c47aaeb16c945ae84ece505aada2a7c P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:01.077734 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.082290 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.085227 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.097151 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:01.101029 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:01.105150 11662 consensus_queue.cc:237] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:01.107038 13185 raft_consensus.cc:1261] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:01.107617 16947 consensus_queue.cc:1035] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:01.108732 11532 raft_consensus.cc:1261] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:01.109195 16947 consensus_queue.cc:1035] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:01.127979 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.128352 17322 raft_consensus.cc:2932] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:01.132874 13185 raft_consensus.cc:2932] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:01.139613 11122 catalog_manager.cc:5484] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 reported cstate change: config changed from index 4 to 5, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:01.141362 11532 raft_consensus.cc:2932] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:01.155930 11642 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c47aaeb16c945ae84ece505aada2a7c with delete_type TABLET_DATA_TOMBSTONED (TS 603919bb02364c04b6c530245b8b02f2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53514
I20240501 08:00:01.156826 11248 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b7de08d2f004cd8abc2bbe1cf7b1444 with delete_type TABLET_DATA_TOMBSTONED (TS e2dafc1866314669ad0de8fb7bf7fd05 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42432
I20240501 08:00:01.162261 17353 tablet_replica.cc:330] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05: stopping tablet replica
I20240501 08:00:01.162638 17353 raft_consensus.cc:2229] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:01.162825 17353 raft_consensus.cc:2258] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:01.163244 17352 tablet_replica.cc:330] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2: stopping tablet replica
I20240501 08:00:01.163517 17352 raft_consensus.cc:2229] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:01.163759 17352 raft_consensus.cc:2258] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:01.164536 17353 ts_tablet_manager.cc:1875] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:01.167670 17352 ts_tablet_manager.cc:1875] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:01.167894 17353 ts_tablet_manager.cc:1888] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:00:01.168006 17353 log.cc:1196] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P e2dafc1866314669ad0de8fb7bf7fd05: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5b7de08d2f004cd8abc2bbe1cf7b1444
I20240501 08:00:01.168673 11113 catalog_manager.cc:4830] TS e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085): tablet 5b7de08d2f004cd8abc2bbe1cf7b1444 (table rebalance_test_table_6 [id=701e50fb54354219a26221f76a7352d2]) successfully deleted
I20240501 08:00:01.171340 17352 ts_tablet_manager.cc:1888] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:01.171550 17352 log.cc:1196] T 5c47aaeb16c945ae84ece505aada2a7c P 603919bb02364c04b6c530245b8b02f2: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5c47aaeb16c945ae84ece505aada2a7c
I20240501 08:00:01.173981 11112 catalog_manager.cc:4830] TS 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): tablet 5c47aaeb16c945ae84ece505aada2a7c (table rebalance_test_table_8 [id=df5f9864bbf8499dbaa392bcb85701c0]) successfully deleted
I20240501 08:00:01.180179 11401 consensus_queue.cc:237] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:01.181807 11270 raft_consensus.cc:1261] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:01.182314 17307 consensus_queue.cc:1035] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:01.182845 13054 raft_consensus.cc:1261] T 164fd265561e47128e7ff585b8cfab07 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:01.183845 13448 raft_consensus.cc:1261] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:01.184764 15127 consensus_queue.cc:1035] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:01.185094 17307 consensus_queue.cc:1035] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:01.187963 15127 raft_consensus.cc:2932] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:01.193931 13448 raft_consensus.cc:2932] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:01.195256 13054 raft_consensus.cc:2932] T 164fd265561e47128e7ff585b8cfab07 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:01.196659 11127 catalog_manager.cc:5484] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:01.197360 11270 raft_consensus.cc:2932] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:01.203770 11510 tablet_service.cc:1514] Processing DeleteTablet for tablet 164fd265561e47128e7ff585b8cfab07 with delete_type TABLET_DATA_TOMBSTONED (TS 22ec12774a4047a8b859743dc73f8ff6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52810
I20240501 08:00:01.207513 17421 tablet_replica.cc:330] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: stopping tablet replica
I20240501 08:00:01.207693 17421 raft_consensus.cc:2229] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:01.207854 17421 raft_consensus.cc:2258] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:01.208726 17421 ts_tablet_manager.cc:1875] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:01.213092 17421 ts_tablet_manager.cc:1888] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:01.213173 17421 log.cc:1196] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/164fd265561e47128e7ff585b8cfab07
I20240501 08:00:01.214221 11114 catalog_manager.cc:4830] TS 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957): tablet 164fd265561e47128e7ff585b8cfab07 (table rebalance_test_table_6 [id=701e50fb54354219a26221f76a7352d2]) successfully deleted
I20240501 08:00:01.257467 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.259461 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.275872 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:01.285930 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.286947 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:01.289762 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.295835 17307 raft_consensus.cc:979] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835: : Instructing follower e2dafc1866314669ad0de8fb7bf7fd05 to start an election
I20240501 08:00:01.296051 17307 raft_consensus.cc:1067] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Signalling peer e2dafc1866314669ad0de8fb7bf7fd05 to start an election
I20240501 08:00:01.296766 11270 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9431cf5853c4148ac38855b1c083d4f"
dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
from {username='slave'} at 127.0.99.130:38885
I20240501 08:00:01.296953 11270 raft_consensus.cc:482] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:01.297016 11270 raft_consensus.cc:3037] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:01.298086 11270 raft_consensus.cc:504] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:01.299523 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9431cf5853c4148ac38855b1c083d4f" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 08:00:01.299647 11401 raft_consensus.cc:3032] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:01.299685 11401 raft_consensus.cc:729] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Running, Role: LEADER
I20240501 08:00:01.299834 11401 consensus_queue.cc:260] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:01.300005 11401 raft_consensus.cc:3037] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:01.301020 11401 raft_consensus.cc:2454] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 2.
I20240501 08:00:01.303326 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.311451 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9431cf5853c4148ac38855b1c083d4f" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ca177138f2c44d2eafa1646b795f7608"
I20240501 08:00:01.311619 13448 raft_consensus.cc:3037] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:01.312662 13448 raft_consensus.cc:2454] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 2.
I20240501 08:00:01.313050 11206 leader_election.cc:302] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835, ca177138f2c44d2eafa1646b795f7608, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 08:00:01.313292 17009 raft_consensus.cc:2781] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:01.313362 17009 raft_consensus.cc:686] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 LEADER]: Becoming Leader. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Running, Role: LEADER
I20240501 08:00:01.313515 17009 consensus_queue.cc:237] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:01.315027 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9431cf5853c4148ac38855b1c083d4f" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
I20240501 08:00:01.315176 11532 raft_consensus.cc:3037] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:01.316156 11532 raft_consensus.cc:2454] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 2.
I20240501 08:00:01.317028 11270 leader_election.cc:288] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 2 election: Requested vote from peers 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957), ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503)
I20240501 08:00:01.317489 11127 catalog_manager.cc:5484] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: term changed from 1 to 2, leader changed from 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) to e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:01.322926 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:01.323725 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:01.326067 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:01.341672 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.348392 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:01.349326 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.364231 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.373165 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:01.383786 11248 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 08:00:01.409587 17486 log.cc:924] Time spent T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20240501 08:00:01.464725 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:01.472239 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.480434 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:01.483821 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.492161 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:01.493456 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:01.502604 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.509919 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.519577 11248 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:00:01.521430 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:01.524796 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.538133 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.539618 11248 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:00:01.562844 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:01.572630 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.576136 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.617372 11401 consensus_queue.cc:237] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:01.618184 11270 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bd8348990f1b496bb62d8c0484193442"
dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43396
I20240501 08:00:01.618346 11270 raft_consensus.cc:595] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:01.619014 13054 raft_consensus.cc:1261] T 64613ea537c842f29d810e829278394e P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:01.620118 13316 raft_consensus.cc:1261] T 64613ea537c842f29d810e829278394e P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:01.620761 17307 consensus_queue.cc:1035] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:01.621812 11662 raft_consensus.cc:1261] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:01.623025 17295 consensus_queue.cc:1035] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:01.623390 17295 consensus_queue.cc:1035] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:01.627224 17307 raft_consensus.cc:2932] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:01.630306 13054 raft_consensus.cc:2932] T 64613ea537c842f29d810e829278394e P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:01.631772 11121 catalog_manager.cc:5484] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:01.632346 11662 raft_consensus.cc:2932] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:01.636443 13316 raft_consensus.cc:2932] T 64613ea537c842f29d810e829278394e P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:01.641451 11510 tablet_service.cc:1514] Processing DeleteTablet for tablet 64613ea537c842f29d810e829278394e with delete_type TABLET_DATA_TOMBSTONED (TS 22ec12774a4047a8b859743dc73f8ff6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52810
I20240501 08:00:01.641705 17421 tablet_replica.cc:330] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6: stopping tablet replica
I20240501 08:00:01.641856 17421 raft_consensus.cc:2229] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:01.641991 17421 raft_consensus.cc:2258] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:01.642585 17421 ts_tablet_manager.cc:1875] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:01.644784 17421 ts_tablet_manager.cc:1888] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:01.644855 17421 log.cc:1196] T 64613ea537c842f29d810e829278394e P 22ec12774a4047a8b859743dc73f8ff6: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/64613ea537c842f29d810e829278394e
I20240501 08:00:01.645372 11114 catalog_manager.cc:4830] TS 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957): tablet 64613ea537c842f29d810e829278394e (table rebalance_test_table_1 [id=98fb02bdb5f54b1699749e3c8d588a5e]) successfully deleted
I20240501 08:00:01.667026 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.671532 11510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:01.675360 11662 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b21b6e2023e842739a7c0cc946e7c2fd"
dest_uuid: "603919bb02364c04b6c530245b8b02f2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52502
I20240501 08:00:01.675567 11662 raft_consensus.cc:595] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:01.683333 11248 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:00:01.684967 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.686581 16396 raft_consensus.cc:979] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05: : Instructing follower 22ec12774a4047a8b859743dc73f8ff6 to start an election
I20240501 08:00:01.686693 16396 raft_consensus.cc:1067] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Signalling peer 22ec12774a4047a8b859743dc73f8ff6 to start an election
I20240501 08:00:01.687325 11532 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bd8348990f1b496bb62d8c0484193442"
dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
from {username='slave'} at 127.0.99.129:40097
I20240501 08:00:01.687417 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:01.687461 11532 raft_consensus.cc:482] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:01.687511 11532 raft_consensus.cc:3037] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:01.688542 11532 raft_consensus.cc:504] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:01.688979 11532 leader_election.cc:288] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 2 election: Requested vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
I20240501 08:00:01.689532 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8348990f1b496bb62d8c0484193442" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e5c03261d31c4d9a91077a17e1cd0336"
I20240501 08:00:01.689714 13316 raft_consensus.cc:3037] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:01.690817 13316 raft_consensus.cc:2454] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 2.
I20240501 08:00:01.691715 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8348990f1b496bb62d8c0484193442" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 08:00:01.691833 11662 raft_consensus.cc:3037] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:01.692824 11662 raft_consensus.cc:2454] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 2.
I20240501 08:00:01.693603 11465 leader_election.cc:302] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 603919bb02364c04b6c530245b8b02f2, e5c03261d31c4d9a91077a17e1cd0336; no voters:
I20240501 08:00:01.693786 17595 raft_consensus.cc:2781] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:01.693845 17595 raft_consensus.cc:686] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Becoming Leader. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Running, Role: LEADER
I20240501 08:00:01.693993 17595 consensus_queue.cc:237] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:01.695423 11270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8348990f1b496bb62d8c0484193442" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 08:00:01.695554 11270 raft_consensus.cc:3032] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:01.695590 11270 raft_consensus.cc:729] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Running, Role: LEADER
I20240501 08:00:01.695757 11270 consensus_queue.cc:260] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:01.695950 11270 raft_consensus.cc:3037] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:01.696045 11127 catalog_manager.cc:5484] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: term changed from 1 to 2, leader changed from e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) to 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:01.696957 11270 raft_consensus.cc:2454] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 2.
I20240501 08:00:01.720484 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.727891 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.754611 11642 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:01.757755 17333 raft_consensus.cc:979] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: : Instructing follower e2dafc1866314669ad0de8fb7bf7fd05 to start an election
I20240501 08:00:01.757854 17333 raft_consensus.cc:1067] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Signalling peer e2dafc1866314669ad0de8fb7bf7fd05 to start an election
I20240501 08:00:01.758615 11270 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b21b6e2023e842739a7c0cc946e7c2fd"
dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
from {username='slave'} at 127.0.99.132:38941
I20240501 08:00:01.758772 11270 raft_consensus.cc:482] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:01.758833 11270 raft_consensus.cc:3037] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:01.759898 11270 raft_consensus.cc:504] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:01.760390 11270 leader_election.cc:288] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 2 election: Requested vote from peers 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
I20240501 08:00:01.760890 11400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21b6e2023e842739a7c0cc946e7c2fd" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 08:00:01.761011 11400 raft_consensus.cc:3037] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:01.761144 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21b6e2023e842739a7c0cc946e7c2fd" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "60e8857abe7c41038af46ebb5a095a90"
I20240501 08:00:01.761330 13054 raft_consensus.cc:3037] T b21b6e2023e842739a7c0cc946e7c2fd P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:01.762022 11400 raft_consensus.cc:2454] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 2.
I20240501 08:00:01.762369 13054 raft_consensus.cc:2454] T b21b6e2023e842739a7c0cc946e7c2fd P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 2.
I20240501 08:00:01.762902 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21b6e2023e842739a7c0cc946e7c2fd" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 08:00:01.763027 11662 raft_consensus.cc:3032] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:01.763063 11662 raft_consensus.cc:729] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Running, Role: LEADER
I20240501 08:00:01.763029 11205 leader_election.cc:302] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60e8857abe7c41038af46ebb5a095a90, 655ebda1fb2c49c4af7087a760529835, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 08:00:01.763216 17009 raft_consensus.cc:2781] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:01.763278 17009 raft_consensus.cc:686] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 LEADER]: Becoming Leader. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Running, Role: LEADER
I20240501 08:00:01.763240 11662 consensus_queue.cc:260] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:01.763411 17009 consensus_queue.cc:237] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:01.763453 11662 raft_consensus.cc:3037] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:01.764410 11662 raft_consensus.cc:2454] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 2.
I20240501 08:00:01.767843 11122 catalog_manager.cc:5484] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: term changed from 1 to 2, leader changed from 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) to e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:01.777758 13448 raft_consensus.cc:1261] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:00:01.778712 16396 consensus_queue.cc:1035] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:01.783362 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.791751 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.792225 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:01.796865 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.800423 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.818799 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:01.838791 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:01.848044 11532 raft_consensus.cc:1261] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:00:01.848628 17009 consensus_queue.cc:1035] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:01.849416 11400 raft_consensus.cc:1261] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:00:01.850071 16396 consensus_queue.cc:1035] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:01.852288 11248 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:00:01.882686 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.886224 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:01.888154 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:01.923857 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.925591 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.928050 11248 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:00:01.934607 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:01.956548 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:01.980063 11400 consensus_queue.cc:237] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:01.981825 13316 raft_consensus.cc:1261] T 5c47aaeb16c945ae84ece505aada2a7c P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:01.981882 13446 raft_consensus.cc:1261] T 5c47aaeb16c945ae84ece505aada2a7c P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:01.982369 17027 consensus_queue.cc:1035] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:01.982784 17027 consensus_queue.cc:1035] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:01.985363 17607 raft_consensus.cc:2932] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:01.985634 11663 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53193305f868433c852755e8a8a81766"
dest_uuid: "603919bb02364c04b6c530245b8b02f2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52536
I20240501 08:00:01.985774 11663 raft_consensus.cc:595] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:01.988445 11122 catalog_manager.cc:5484] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 6 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:01.988624 13316 raft_consensus.cc:2932] T 5c47aaeb16c945ae84ece505aada2a7c P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:01.989344 13446 raft_consensus.cc:2932] T 5c47aaeb16c945ae84ece505aada2a7c P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:01.996109 11248 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c47aaeb16c945ae84ece505aada2a7c with delete_type TABLET_DATA_TOMBSTONED (TS e2dafc1866314669ad0de8fb7bf7fd05 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42432
I20240501 08:00:01.998801 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:02.000536 17792 tablet_replica.cc:330] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05: stopping tablet replica
I20240501 08:00:02.000861 17792 raft_consensus.cc:2229] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:02.001160 17792 raft_consensus.cc:2258] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:02.003537 17792 ts_tablet_manager.cc:1875] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:02.005762 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.008119 17792 ts_tablet_manager.cc:1888] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:00:02.008301 17792 log.cc:1196] T 5c47aaeb16c945ae84ece505aada2a7c P e2dafc1866314669ad0de8fb7bf7fd05: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5c47aaeb16c945ae84ece505aada2a7c
I20240501 08:00:02.009001 11113 catalog_manager.cc:4830] TS e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085): tablet 5c47aaeb16c945ae84ece505aada2a7c (table rebalance_test_table_8 [id=df5f9864bbf8499dbaa392bcb85701c0]) successfully deleted
I20240501 08:00:02.014395 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.015578 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:02.020084 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:02.028048 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.028880 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.034271 17678 raft_consensus.cc:979] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2: : Instructing follower 655ebda1fb2c49c4af7087a760529835 to start an election
I20240501 08:00:02.034389 17678 raft_consensus.cc:1067] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Signalling peer 655ebda1fb2c49c4af7087a760529835 to start an election
I20240501 08:00:02.034901 11400 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53193305f868433c852755e8a8a81766"
dest_uuid: "655ebda1fb2c49c4af7087a760529835"
from {username='slave'} at 127.0.99.132:45609
I20240501 08:00:02.035053 11400 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:02.035104 11400 raft_consensus.cc:3037] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:02.036168 11400 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:02.036629 11400 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 2 election: Requested vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503)
I20240501 08:00:02.036932 11270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53193305f868433c852755e8a8a81766" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 08:00:02.036964 11663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53193305f868433c852755e8a8a81766" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 08:00:02.037066 11270 raft_consensus.cc:3037] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:02.037106 11663 raft_consensus.cc:3032] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:02.037146 11663 raft_consensus.cc:729] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Running, Role: LEADER
I20240501 08:00:02.037307 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53193305f868433c852755e8a8a81766" candidate_uuid: "655ebda1fb2c49c4af7087a760529835" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ca177138f2c44d2eafa1646b795f7608"
I20240501 08:00:02.037309 11663 consensus_queue.cc:260] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:02.037420 13446 raft_consensus.cc:3037] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:02.037528 11663 raft_consensus.cc:3037] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:02.038137 11270 raft_consensus.cc:2454] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 2.
I20240501 08:00:02.038352 13446 raft_consensus.cc:2454] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 2.
I20240501 08:00:02.038684 11337 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835, ca177138f2c44d2eafa1646b795f7608, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 08:00:02.038875 17027 raft_consensus.cc:2781] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:02.038956 17027 raft_consensus.cc:686] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 2 LEADER]: Becoming Leader. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Running, Role: LEADER
I20240501 08:00:02.039106 17027 consensus_queue.cc:237] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:02.039765 11663 raft_consensus.cc:2454] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655ebda1fb2c49c4af7087a760529835 in term 2.
I20240501 08:00:02.041328 11127 catalog_manager.cc:5484] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: term changed from 1 to 2, leader changed from 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) to 655ebda1fb2c49c4af7087a760529835 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:02.071954 11248 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:00:02.099085 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:02.104496 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.108273 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.121176 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:02.121762 11270 raft_consensus.cc:1261] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:00:02.122195 17595 consensus_queue.cc:1035] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:02.139359 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:02.170271 11663 raft_consensus.cc:1261] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:00:02.170723 17595 consensus_queue.cc:1035] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:02.175760 11400 raft_consensus.cc:1261] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:00:02.176333 17714 consensus_queue.cc:1035] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:02.179584 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.187695 11248 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:00:02.193611 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.200438 13316 raft_consensus.cc:1261] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:00:02.200917 16360 consensus_queue.cc:1035] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:02.216027 11663 raft_consensus.cc:1261] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:00:02.216509 17714 consensus_queue.cc:1035] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:02.219810 13054 raft_consensus.cc:1261] T b21b6e2023e842739a7c0cc946e7c2fd P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:00:02.220247 16396 consensus_queue.cc:1035] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:02.222561 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:02.234143 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:02.234400 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.238222 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.251071 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.264495 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:02.267076 11248 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:00:02.302286 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.351641 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.352489 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:02.353183 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.354362 11248 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:00:02.356282 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.358378 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:02.366992 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.378172 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:02.392688 11532 consensus_queue.cc:237] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:02.394503 11400 raft_consensus.cc:1261] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:02.394519 13446 raft_consensus.cc:1261] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:02.395004 16360 consensus_queue.cc:1035] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:02.395224 17897 consensus_queue.cc:1035] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:02.398936 17897 raft_consensus.cc:2932] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:02.399366 13446 raft_consensus.cc:2932] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:02.401836 11124 catalog_manager.cc:5484] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 reported cstate change: config changed from index 4 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:02.404873 11642 tablet_service.cc:1514] Processing DeleteTablet for tablet 90f11b5693d7456d8a604e6c8726e69a with delete_type TABLET_DATA_TOMBSTONED (TS 603919bb02364c04b6c530245b8b02f2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53514
I20240501 08:00:02.407505 17974 tablet_replica.cc:330] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: stopping tablet replica
I20240501 08:00:02.407685 17974 raft_consensus.cc:2229] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:02.408380 17974 raft_consensus.cc:2258] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:02.409210 17974 ts_tablet_manager.cc:1875] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:02.411571 17974 ts_tablet_manager.cc:1888] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:00:02.411641 17974 log.cc:1196] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/90f11b5693d7456d8a604e6c8726e69a
I20240501 08:00:02.412117 11112 catalog_manager.cc:4830] TS 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): tablet 90f11b5693d7456d8a604e6c8726e69a (table rebalance_test_table_5 [id=0af21d4c4fcb49a7ac0756a5ded5d57d]) successfully deleted
I20240501 08:00:02.431759 11400 raft_consensus.cc:2932] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:02.483815 11400 consensus_queue.cc:237] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:02.485432 13054 raft_consensus.cc:1261] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:02.485450 11532 raft_consensus.cc:1261] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:02.486054 17027 consensus_queue.cc:1035] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:02.486407 17027 consensus_queue.cc:1035] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 08:00:02.504949 17027 raft_consensus.cc:2932] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:02.506492 11532 raft_consensus.cc:2932] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:02.509220 13054 raft_consensus.cc:2932] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:02.512022 11127 catalog_manager.cc:5484] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 4 to 5, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:02.518563 11642 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e8550ea95984d08937b9b3e2af12bec with delete_type TABLET_DATA_TOMBSTONED (TS 603919bb02364c04b6c530245b8b02f2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:53514
I20240501 08:00:02.518884 17974 tablet_replica.cc:330] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: stopping tablet replica
I20240501 08:00:02.519026 17974 raft_consensus.cc:2229] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:02.519179 17974 raft_consensus.cc:2258] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:02.519811 17974 ts_tablet_manager.cc:1875] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:02.522125 17974 ts_tablet_manager.cc:1888] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:00:02.522193 17974 log.cc:1196] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5e8550ea95984d08937b9b3e2af12bec
I20240501 08:00:02.525501 11112 catalog_manager.cc:4830] TS 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): tablet 5e8550ea95984d08937b9b3e2af12bec (table rebalance_test_table_7 [id=36e20074d8dc4402bd7d9b2039213af5]) successfully deleted
I20240501 08:00:02.535163 11270 raft_consensus.cc:1261] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:00:02.535786 17295 consensus_queue.cc:1035] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:02.540030 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:02.555480 11248 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:00:02.558017 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.558655 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:02.558656 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.572082 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.586252 11662 raft_consensus.cc:1261] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:00:02.586752 17295 consensus_queue.cc:1035] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:02.609949 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:02.620055 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:02.620589 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.623800 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:02.624428 13446 raft_consensus.cc:1261] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:00:02.627111 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.630388 17027 consensus_queue.cc:1035] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:02.631951 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.635125 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:02.647725 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.651752 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:02.655036 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.658135 11248 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:00:02.663606 11248 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:00:02.696102 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.697679 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.704334 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:02.708132 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.723740 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:02.758083 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.771777 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:02.775017 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.782562 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.784044 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.787962 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.796994 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:02.806664 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:02.836100 11248 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:00:02.884020 11400 consensus_queue.cc:237] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:02.885782 13316 raft_consensus.cc:1261] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:02.886713 17027 consensus_queue.cc:1035] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:02.890964 17027 raft_consensus.cc:2932] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:02.893648 13316 raft_consensus.cc:2932] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:02.896538 11127 catalog_manager.cc:5484] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 5 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:02.899771 11642 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c23e3bd7ec34876956046e5d32e4d82 with delete_type TABLET_DATA_TOMBSTONED (TS 603919bb02364c04b6c530245b8b02f2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53514
I20240501 08:00:02.900049 17974 tablet_replica.cc:330] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: stopping tablet replica
I20240501 08:00:02.900219 17974 raft_consensus.cc:2229] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:02.900421 17974 raft_consensus.cc:2258] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:02.901013 17974 ts_tablet_manager.cc:1875] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:02.903980 17974 ts_tablet_manager.cc:1888] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:02.904062 17974 log.cc:1196] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7c23e3bd7ec34876956046e5d32e4d82
I20240501 08:00:02.904053 11400 consensus_queue.cc:237] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:02.904606 11112 catalog_manager.cc:4830] TS 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): tablet 7c23e3bd7ec34876956046e5d32e4d82 (table rebalance_test_table_9 [id=1deda0b4866b43c9ba7c69dae5d65e4a]) successfully deleted
I20240501 08:00:02.906325 13316 raft_consensus.cc:1261] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:02.906848 18099 consensus_queue.cc:1035] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:02.908032 11270 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a24be92aec7544089c7818f2802b9c85"
dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43552
I20240501 08:00:02.908183 11270 raft_consensus.cc:595] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:02.909360 11532 consensus_queue.cc:237] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:02.909727 18099 raft_consensus.cc:2932] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:02.910729 13316 raft_consensus.cc:2932] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:02.912225 11270 raft_consensus.cc:1261] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:02.912693 13316 raft_consensus.cc:1261] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:02.912660 16360 consensus_queue.cc:1035] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:02.913086 16360 consensus_queue.cc:1035] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:02.915127 16360 raft_consensus.cc:2932] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:02.915599 11269 raft_consensus.cc:2932] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:02.915692 11270 consensus_queue.cc:237] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:02.918601 13054 raft_consensus.cc:1261] T c95c4f3ef08f44d1b69fb7b98426f5af P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:02.919075 16396 consensus_queue.cc:1035] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:02.919157 11126 catalog_manager.cc:5484] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index 4 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:02.919309 11532 raft_consensus.cc:1261] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:02.920266 16396 consensus_queue.cc:1035] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:02.920236 13316 raft_consensus.cc:2932] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:02.922251 11124 catalog_manager.cc:5484] T 7c23e3bd7ec34876956046e5d32e4d82 P e5c03261d31c4d9a91077a17e1cd0336 reported cstate change: config changed from index 6 to 7, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
W20240501 08:00:02.927546 11334 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771): Couldn't send request to peer 53d1d2c0767b4f6195b69758a3dd6cf9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c23e3bd7ec34876956046e5d32e4d82. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:02.930783 18263 raft_consensus.cc:2932] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:02.931301 11112 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 7c23e3bd7ec34876956046e5d32e4d82 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:00:02.931943 11642 tablet_service.cc:1514] Processing DeleteTablet for tablet bd8348990f1b496bb62d8c0484193442 with delete_type TABLET_DATA_TOMBSTONED (TS 603919bb02364c04b6c530245b8b02f2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53514
I20240501 08:00:02.932515 13054 raft_consensus.cc:2932] T c95c4f3ef08f44d1b69fb7b98426f5af P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:02.937870 11532 raft_consensus.cc:2932] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:02.939731 17974 tablet_replica.cc:330] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: stopping tablet replica
I20240501 08:00:02.939915 17974 raft_consensus.cc:2229] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:02.940074 17974 raft_consensus.cc:2258] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:02.941895 17974 ts_tablet_manager.cc:1875] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:02.943199 11128 catalog_manager.cc:5484] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 4 to 5, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:02.945854 17974 ts_tablet_manager.cc:1888] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:00:02.945937 17974 log.cc:1196] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/bd8348990f1b496bb62d8c0484193442
I20240501 08:00:02.949949 11380 tablet_service.cc:1514] Processing DeleteTablet for tablet c95c4f3ef08f44d1b69fb7b98426f5af with delete_type TABLET_DATA_TOMBSTONED (TS 655ebda1fb2c49c4af7087a760529835 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:43354
I20240501 08:00:02.950044 11112 catalog_manager.cc:4830] TS 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): tablet bd8348990f1b496bb62d8c0484193442 (table rebalance_test_table_2 [id=69f109081c7449be8819974c5363fa32]) successfully deleted
I20240501 08:00:02.953171 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:02.960110 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.966246 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.979624 11248 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:00:02.986002 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.988831 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:02.989292 18299 tablet_replica.cc:330] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835: stopping tablet replica
I20240501 08:00:02.989495 18299 raft_consensus.cc:2229] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:02.989723 18299 raft_consensus.cc:2258] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:02.990353 18299 ts_tablet_manager.cc:1875] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:02.994136 18299 ts_tablet_manager.cc:1888] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:00:02.994299 18299 log.cc:1196] T c95c4f3ef08f44d1b69fb7b98426f5af P 655ebda1fb2c49c4af7087a760529835: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c95c4f3ef08f44d1b69fb7b98426f5af
I20240501 08:00:02.995635 11112 catalog_manager.cc:4830] TS 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575): tablet c95c4f3ef08f44d1b69fb7b98426f5af (table rebalance_test_table_0 [id=24b4f6e8cc8c44f6961591cac7beacb4]) successfully deleted
I20240501 08:00:03.004459 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:03.004787 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:03.028259 18263 raft_consensus.cc:979] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05: : Instructing follower 22ec12774a4047a8b859743dc73f8ff6 to start an election
I20240501 08:00:03.028409 17009 raft_consensus.cc:1067] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Signalling peer 22ec12774a4047a8b859743dc73f8ff6 to start an election
I20240501 08:00:03.028861 11532 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a24be92aec7544089c7818f2802b9c85"
dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
from {username='slave'} at 127.0.99.129:40097
I20240501 08:00:03.029009 11532 raft_consensus.cc:482] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:03.029065 11532 raft_consensus.cc:3037] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:03.030164 11532 raft_consensus.cc:504] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:03.030812 11532 leader_election.cc:288] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 2 election: Requested vote from peers 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771)
I20240501 08:00:03.031208 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24be92aec7544089c7818f2802b9c85" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ca177138f2c44d2eafa1646b795f7608"
I20240501 08:00:03.031353 13446 raft_consensus.cc:3037] T a24be92aec7544089c7818f2802b9c85 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:03.031986 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24be92aec7544089c7818f2802b9c85" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9"
I20240501 08:00:03.032162 13185 raft_consensus.cc:3037] T a24be92aec7544089c7818f2802b9c85 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:03.032404 13446 raft_consensus.cc:2454] T a24be92aec7544089c7818f2802b9c85 P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 2.
I20240501 08:00:03.033176 13185 raft_consensus.cc:2454] T a24be92aec7544089c7818f2802b9c85 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 2.
I20240501 08:00:03.033654 11663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24be92aec7544089c7818f2802b9c85" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 08:00:03.033769 11663 raft_consensus.cc:3037] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:03.034601 11269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24be92aec7544089c7818f2802b9c85" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 08:00:03.034715 11269 raft_consensus.cc:3032] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:03.034714 11663 raft_consensus.cc:2454] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 2.
I20240501 08:00:03.034749 11269 raft_consensus.cc:729] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Running, Role: LEADER
I20240501 08:00:03.034880 11269 consensus_queue.cc:260] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:03.035108 11269 raft_consensus.cc:3037] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:03.035437 11465 leader_election.cc:302] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 603919bb02364c04b6c530245b8b02f2, ca177138f2c44d2eafa1646b795f7608; no voters:
I20240501 08:00:03.035787 18260 raft_consensus.cc:2781] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:03.035872 18260 raft_consensus.cc:686] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Becoming Leader. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Running, Role: LEADER
I20240501 08:00:03.036026 11269 raft_consensus.cc:2454] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 2.
I20240501 08:00:03.036010 18260 consensus_queue.cc:237] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:03.038856 11122 catalog_manager.cc:5484] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: term changed from 1 to 2, leader changed from e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) to 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:03.111454 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.115000 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:03.124963 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.125859 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:03.130606 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:03.148813 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.149056 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.159029 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:03.166524 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:03.169999 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.176198 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:03.203135 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:03.206485 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:03.214601 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.219521 11662 consensus_queue.cc:237] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:03.221796 13185 raft_consensus.cc:1261] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:03.222319 17333 consensus_queue.cc:1035] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:03.225000 11532 raft_consensus.cc:1261] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:03.226176 13316 raft_consensus.cc:1261] T f61e70ba02384f81b9ae36b1c65465b9 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:03.228017 18428 consensus_queue.cc:1035] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:03.229527 18429 consensus_queue.cc:1035] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:03.232463 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.238461 18428 raft_consensus.cc:2932] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:03.238950 13185 raft_consensus.cc:2932] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:03.241061 11532 raft_consensus.cc:2932] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:03.241999 11126 catalog_manager.cc:5484] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 reported cstate change: config changed from index 5 to 6, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:03.243386 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.247160 11380 tablet_service.cc:1514] Processing DeleteTablet for tablet f61e70ba02384f81b9ae36b1c65465b9 with delete_type TABLET_DATA_TOMBSTONED (TS 655ebda1fb2c49c4af7087a760529835 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43354
I20240501 08:00:03.247454 18299 tablet_replica.cc:330] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835: stopping tablet replica
I20240501 08:00:03.247592 18299 raft_consensus.cc:2229] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:03.247745 18299 raft_consensus.cc:2258] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:03.248383 18299 ts_tablet_manager.cc:1875] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:03.250895 18299 ts_tablet_manager.cc:1888] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:03.250970 18299 log.cc:1196] T f61e70ba02384f81b9ae36b1c65465b9 P 655ebda1fb2c49c4af7087a760529835: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f61e70ba02384f81b9ae36b1c65465b9
I20240501 08:00:03.251791 11112 catalog_manager.cc:4830] TS 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575): tablet f61e70ba02384f81b9ae36b1c65465b9 (table rebalance_test_table_5 [id=0af21d4c4fcb49a7ac0756a5ded5d57d]) successfully deleted
I20240501 08:00:03.266532 13316 raft_consensus.cc:2932] T f61e70ba02384f81b9ae36b1c65465b9 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:03.313454 11532 consensus_queue.cc:237] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:03.317154 11400 raft_consensus.cc:1261] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:03.317243 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.317576 18260 consensus_queue.cc:1035] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:03.318753 13446 raft_consensus.cc:1261] T b4707b952d174a9bb877ad8bd7a11268 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:03.320132 13054 raft_consensus.cc:1261] T b4707b952d174a9bb877ad8bd7a11268 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:03.320370 18261 consensus_queue.cc:1035] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:03.322018 18261 consensus_queue.cc:1035] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:03.323887 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:03.327023 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:03.337234 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.341899 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.342880 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.343070 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:03.346895 11400 raft_consensus.cc:2932] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:03.351619 11128 catalog_manager.cc:5484] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 5 to 6, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:03.346343 18261 raft_consensus.cc:2932] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:03.356184 13054 raft_consensus.cc:2932] T b4707b952d174a9bb877ad8bd7a11268 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:03.360978 11248 tablet_service.cc:1514] Processing DeleteTablet for tablet b4707b952d174a9bb877ad8bd7a11268 with delete_type TABLET_DATA_TOMBSTONED (TS e2dafc1866314669ad0de8fb7bf7fd05 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42432
I20240501 08:00:03.364825 18515 tablet_replica.cc:330] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05: stopping tablet replica
I20240501 08:00:03.365058 18515 raft_consensus.cc:2229] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:03.365279 18515 raft_consensus.cc:2258] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:03.365891 18515 ts_tablet_manager.cc:1875] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:03.370345 18515 ts_tablet_manager.cc:1888] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:03.370539 18515 log.cc:1196] T b4707b952d174a9bb877ad8bd7a11268 P e2dafc1866314669ad0de8fb7bf7fd05: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b4707b952d174a9bb877ad8bd7a11268
I20240501 08:00:03.371131 11113 catalog_manager.cc:4830] TS e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085): tablet b4707b952d174a9bb877ad8bd7a11268 (table rebalance_test_table_4 [id=384b2b245c6b4931a4fc02c6b5d647b2]) successfully deleted
I20240501 08:00:03.371516 13446 raft_consensus.cc:2932] T b4707b952d174a9bb877ad8bd7a11268 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:03.376897 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:03.428700 11269 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "25cee7e51c7c438aa721d71fb82dc71e"
dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43606
I20240501 08:00:03.428879 11269 raft_consensus.cc:595] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:03.443434 18595 ts_tablet_manager.cc:902] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9: Initiating tablet copy from peer 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 08:00:03.451135 11269 raft_consensus.cc:1261] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:03.453471 18595 tablet_copy_client.cc:308] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41575
I20240501 08:00:03.455699 18514 consensus_queue.cc:1035] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:03.464941 11420 tablet_copy_service.cc:140] P 655ebda1fb2c49c4af7087a760529835: Received BeginTabletCopySession request for tablet 7c23e3bd7ec34876956046e5d32e4d82 from peer 53d1d2c0767b4f6195b69758a3dd6cf9 ({username='slave'} at 127.0.99.134:46331)
I20240501 08:00:03.465075 11420 tablet_copy_service.cc:161] P 655ebda1fb2c49c4af7087a760529835: Beginning new tablet copy session on tablet 7c23e3bd7ec34876956046e5d32e4d82 from peer 53d1d2c0767b4f6195b69758a3dd6cf9 at {username='slave'} at 127.0.99.134:46331: session id = 53d1d2c0767b4f6195b69758a3dd6cf9-7c23e3bd7ec34876956046e5d32e4d82
I20240501 08:00:03.466024 11420 tablet_copy_source_session.cc:205] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:03.467203 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c23e3bd7ec34876956046e5d32e4d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:03.469419 18595 tablet_copy_client.cc:783] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:03.469581 18595 tablet_copy_client.cc:647] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:03.480767 18595 tablet_copy_client.cc:523] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:03.484412 18595 tablet_bootstrap.cc:492] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap starting.
I20240501 08:00:03.490695 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:03.504683 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:03.525537 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.527171 13166 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.532810 13446 raft_consensus.cc:1261] T a24be92aec7544089c7818f2802b9c85 P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:03.533529 18261 consensus_queue.cc:1035] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:03.545156 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.548290 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.551254 13185 raft_consensus.cc:1261] T a24be92aec7544089c7818f2802b9c85 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:03.552534 17009 raft_consensus.cc:979] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: : Instructing follower 53d1d2c0767b4f6195b69758a3dd6cf9 to start an election
I20240501 08:00:03.552625 17009 raft_consensus.cc:1067] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Signalling peer 53d1d2c0767b4f6195b69758a3dd6cf9 to start an election
I20240501 08:00:03.553133 13185 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "25cee7e51c7c438aa721d71fb82dc71e"
dest_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9"
from {username='slave'} at 127.0.99.129:58765
I20240501 08:00:03.553332 13185 raft_consensus.cc:482] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:03.553406 13185 raft_consensus.cc:3037] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:03.554455 13185 raft_consensus.cc:504] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:03.556139 13185 leader_election.cc:288] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 2 election: Requested vote from peers 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575), e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
I20240501 08:00:03.556711 11400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25cee7e51c7c438aa721d71fb82dc71e" candidate_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 08:00:03.556833 11400 raft_consensus.cc:3037] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:03.557247 11270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25cee7e51c7c438aa721d71fb82dc71e" candidate_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 08:00:03.557374 11270 raft_consensus.cc:3032] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:03.557415 11270 raft_consensus.cc:729] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Running, Role: LEADER
I20240501 08:00:03.557559 11270 consensus_queue.cc:260] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:03.557801 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:03.557797 11270 raft_consensus.cc:3037] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:03.557868 11400 raft_consensus.cc:2454] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53d1d2c0767b4f6195b69758a3dd6cf9 in term 2.
I20240501 08:00:03.558441 11663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25cee7e51c7c438aa721d71fb82dc71e" candidate_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 08:00:03.558569 11663 raft_consensus.cc:3037] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:03.558817 11270 raft_consensus.cc:2454] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53d1d2c0767b4f6195b69758a3dd6cf9 in term 2.
I20240501 08:00:03.559595 11663 raft_consensus.cc:2454] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53d1d2c0767b4f6195b69758a3dd6cf9 in term 2.
I20240501 08:00:03.559716 11661 raft_consensus.cc:1261] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:03.559669 18514 consensus_queue.cc:1035] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:03.560792 13119 leader_election.cc:302] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9, 655ebda1fb2c49c4af7087a760529835, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 08:00:03.561041 18438 raft_consensus.cc:2781] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:03.561233 18438 raft_consensus.cc:686] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 LEADER]: Becoming Leader. State: Replica: 53d1d2c0767b4f6195b69758a3dd6cf9, State: Running, Role: LEADER
I20240501 08:00:03.561631 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:03.561573 18438 consensus_queue.cc:237] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:03.566691 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25cee7e51c7c438aa721d71fb82dc71e" candidate_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "60e8857abe7c41038af46ebb5a095a90"
I20240501 08:00:03.566841 13054 raft_consensus.cc:3037] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:03.567698 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.567848 13054 raft_consensus.cc:2454] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53d1d2c0767b4f6195b69758a3dd6cf9 in term 2.
I20240501 08:00:03.568516 18261 consensus_queue.cc:1035] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:03.568785 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:03.580375 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.588294 11122 catalog_manager.cc:5484] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 reported cstate change: term changed from 1 to 2, leader changed from e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) to 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:03.596513 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.605924 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:03.607929 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:03.625308 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:03.646752 18595 tablet_bootstrap.cc:492] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:03.647136 18595 tablet_bootstrap.cc:492] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap complete.
I20240501 08:00:03.647334 18595 ts_tablet_manager.cc:1367] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent bootstrapping tablet: real 0.163s user 0.026s sys 0.000s
I20240501 08:00:03.647904 18595 raft_consensus.cc:348] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:03.648027 18595 raft_consensus.cc:729] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d1d2c0767b4f6195b69758a3dd6cf9, State: Initialized, Role: FOLLOWER
I20240501 08:00:03.648234 18595 consensus_queue.cc:260] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:03.648514 18595 ts_tablet_manager.cc:1398] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 08:00:03.652839 11420 tablet_copy_service.cc:342] P 655ebda1fb2c49c4af7087a760529835: Request end of tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-7c23e3bd7ec34876956046e5d32e4d82 received from {username='slave'} at 127.0.99.134:46331
I20240501 08:00:03.652935 11420 tablet_copy_service.cc:433] P 655ebda1fb2c49c4af7087a760529835: ending tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-7c23e3bd7ec34876956046e5d32e4d82 on tablet 7c23e3bd7ec34876956046e5d32e4d82 with peer 53d1d2c0767b4f6195b69758a3dd6cf9
I20240501 08:00:03.655218 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:03.708398 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:03.708659 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:03.712806 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.732251 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.741729 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:03.743808 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:03.755286 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.763278 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:03.773967 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:03.780459 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:03.785637 11400 consensus_queue.cc:237] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:03.787518 11662 raft_consensus.cc:1261] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:03.787524 13316 raft_consensus.cc:1261] T a1ebf29682024645978b3caa2d4988f3 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:03.787680 13054 raft_consensus.cc:1261] T a1ebf29682024645978b3caa2d4988f3 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:03.787976 17295 consensus_queue.cc:1035] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:03.788321 17295 consensus_queue.cc:1035] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:03.788688 17027 consensus_queue.cc:1035] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:03.799208 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:03.801566 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.802212 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:03.823092 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.842386 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:03.847632 17027 raft_consensus.cc:2932] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:03.849946 11662 raft_consensus.cc:2932] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:03.850942 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.852326 13054 raft_consensus.cc:2932] T a1ebf29682024645978b3caa2d4988f3 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:03.855165 11120 catalog_manager.cc:5484] T a1ebf29682024645978b3caa2d4988f3 P 60e8857abe7c41038af46ebb5a095a90 reported cstate change: config changed from index 5 to 6, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:03.860402 11248 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ebf29682024645978b3caa2d4988f3 with delete_type TABLET_DATA_TOMBSTONED (TS e2dafc1866314669ad0de8fb7bf7fd05 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42432
I20240501 08:00:03.860657 18515 tablet_replica.cc:330] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05: stopping tablet replica
I20240501 08:00:03.860824 18515 raft_consensus.cc:2229] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:03.861021 18515 raft_consensus.cc:2258] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:03.861685 18515 ts_tablet_manager.cc:1875] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:03.863956 18515 ts_tablet_manager.cc:1888] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:03.864038 18515 log.cc:1196] T a1ebf29682024645978b3caa2d4988f3 P e2dafc1866314669ad0de8fb7bf7fd05: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a1ebf29682024645978b3caa2d4988f3
I20240501 08:00:03.864549 11113 catalog_manager.cc:4830] TS e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085): tablet a1ebf29682024645978b3caa2d4988f3 (table rebalance_test_table_0 [id=24b4f6e8cc8c44f6961591cac7beacb4]) successfully deleted
I20240501 08:00:03.865729 13316 raft_consensus.cc:2932] T a1ebf29682024645978b3caa2d4988f3 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:03.899117 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.899821 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:03.900748 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.900998 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:03.901016 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:03.902266 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:03.903688 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:03.920856 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:03.958529 11532 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "17bb1ff829bc47a6a0bd6030517652aa"
dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56508
I20240501 08:00:03.958738 11532 raft_consensus.cc:595] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:03.977730 13185 raft_consensus.cc:1203] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:00:04.038399 11400 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer 53d1d2c0767b4f6195b69758a3dd6cf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:04.039160 18664 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:04.051411 11662 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Refusing update from remote peer 53d1d2c0767b4f6195b69758a3dd6cf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:04.051882 18664 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:04.059096 11269 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Refusing update from remote peer 53d1d2c0767b4f6195b69758a3dd6cf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:04.059638 18664 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:04.068946 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:04.071766 13053 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Refusing update from remote peer 53d1d2c0767b4f6195b69758a3dd6cf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:04.072238 18943 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:04.077251 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.077527 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:04.078729 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.081562 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.096582 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.103673 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.108855 16360 raft_consensus.cc:979] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6: : Instructing follower e2dafc1866314669ad0de8fb7bf7fd05 to start an election
I20240501 08:00:04.108989 16360 raft_consensus.cc:1067] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Signalling peer e2dafc1866314669ad0de8fb7bf7fd05 to start an election
I20240501 08:00:04.109524 11268 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "17bb1ff829bc47a6a0bd6030517652aa"
dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
from {username='slave'} at 127.0.99.131:48113
I20240501 08:00:04.109660 11268 raft_consensus.cc:482] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:04.109714 11268 raft_consensus.cc:3037] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:04.111076 11268 raft_consensus.cc:504] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:04.111660 11268 leader_election.cc:288] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 2 election: Requested vote from peers 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575), 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957), 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771)
I20240501 08:00:04.111999 11400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17bb1ff829bc47a6a0bd6030517652aa" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 08:00:04.112119 11400 raft_consensus.cc:3037] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:04.112540 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17bb1ff829bc47a6a0bd6030517652aa" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
I20240501 08:00:04.112663 11532 raft_consensus.cc:3032] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:04.112708 11532 raft_consensus.cc:729] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Running, Role: LEADER
I20240501 08:00:04.112879 11532 consensus_queue.cc:260] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:04.112979 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17bb1ff829bc47a6a0bd6030517652aa" candidate_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9"
I20240501 08:00:04.113095 11532 raft_consensus.cc:3037] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:04.113099 13185 raft_consensus.cc:3037] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:04.113157 11400 raft_consensus.cc:2454] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 2.
I20240501 08:00:04.114092 13185 raft_consensus.cc:2454] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 2.
I20240501 08:00:04.114153 11532 raft_consensus.cc:2454] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dafc1866314669ad0de8fb7bf7fd05 in term 2.
I20240501 08:00:04.114485 11203 leader_election.cc:302] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9, 655ebda1fb2c49c4af7087a760529835, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 08:00:04.114657 18263 raft_consensus.cc:2781] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:04.114729 18263 raft_consensus.cc:686] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 LEADER]: Becoming Leader. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Running, Role: LEADER
I20240501 08:00:04.114863 18263 consensus_queue.cc:237] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:04.117305 11121 catalog_manager.cc:5484] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: term changed from 1 to 2, leader changed from 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) to e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:04.184092 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:04.193027 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.209276 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.209362 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:04.216348 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.225878 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.226145 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:04.229781 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:04.237543 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.259449 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.259615 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.260334 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:04.260807 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.274966 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.297339 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.297906 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.298822 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:04.310803 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.312270 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:04.314934 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:04.319993 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:04.346261 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:04.368456 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.382241 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.387111 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.424443 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:04.429086 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.430006 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:04.431641 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.436399 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.440615 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.445436 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:04.466676 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.542035 11269 consensus_queue.cc:237] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:04.544047 13185 raft_consensus.cc:1261] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:04.544384 11400 raft_consensus.cc:1261] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:04.544694 18953 consensus_queue.cc:1035] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:04.545009 18991 consensus_queue.cc:1035] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:04.547174 18991 raft_consensus.cc:2932] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:04.553434 11400 raft_consensus.cc:2932] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:04.556545 11121 catalog_manager.cc:5484] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 4 to 5, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:04.557096 13185 raft_consensus.cc:2932] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:04.561784 11510 tablet_service.cc:1514] Processing DeleteTablet for tablet ea97f34590684bcf998228245dbcd9b0 with delete_type TABLET_DATA_TOMBSTONED (TS 22ec12774a4047a8b859743dc73f8ff6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52810
I20240501 08:00:04.562812 19149 tablet_replica.cc:330] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6: stopping tablet replica
I20240501 08:00:04.562966 19149 raft_consensus.cc:2229] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:04.563122 19149 raft_consensus.cc:2258] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:04.563700 19149 ts_tablet_manager.cc:1875] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:04.567107 19149 ts_tablet_manager.cc:1888] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:00:04.567241 19149 log.cc:1196] T ea97f34590684bcf998228245dbcd9b0 P 22ec12774a4047a8b859743dc73f8ff6: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ea97f34590684bcf998228245dbcd9b0
I20240501 08:00:04.568037 11662 consensus_queue.cc:237] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:04.568178 11114 catalog_manager.cc:4830] TS 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957): tablet ea97f34590684bcf998228245dbcd9b0 (table rebalance_test_table_6 [id=701e50fb54354219a26221f76a7352d2]) successfully deleted
I20240501 08:00:04.569339 13185 raft_consensus.cc:1261] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:04.569763 18437 consensus_queue.cc:1035] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:04.574558 18437 raft_consensus.cc:2932] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:04.577117 13185 raft_consensus.cc:2932] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:04.577898 11121 catalog_manager.cc:5484] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 reported cstate change: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:04.581377 11510 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b7de08d2f004cd8abc2bbe1cf7b1444 with delete_type TABLET_DATA_TOMBSTONED (TS 22ec12774a4047a8b859743dc73f8ff6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52810
I20240501 08:00:04.581604 19149 tablet_replica.cc:330] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: stopping tablet replica
I20240501 08:00:04.581737 19149 raft_consensus.cc:2229] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:04.581701 11662 consensus_queue.cc:237] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:04.581884 19149 raft_consensus.cc:2258] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:04.582525 19149 ts_tablet_manager.cc:1875] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 08:00:04.583352 11599 consensus_peers.cc:485] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b7de08d2f004cd8abc2bbe1cf7b1444. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:04.586913 13185 raft_consensus.cc:1261] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:04.587512 19179 consensus_queue.cc:1035] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:04.588675 19149 ts_tablet_manager.cc:1888] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:04.588827 19149 log.cc:1196] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/5b7de08d2f004cd8abc2bbe1cf7b1444
I20240501 08:00:04.589433 11114 catalog_manager.cc:4830] TS 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957): tablet 5b7de08d2f004cd8abc2bbe1cf7b1444 (table rebalance_test_table_6 [id=701e50fb54354219a26221f76a7352d2]) successfully deleted
I20240501 08:00:04.589488 19179 raft_consensus.cc:2932] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:04.589880 11532 consensus_queue.cc:237] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:04.591459 13186 raft_consensus.cc:1261] T b4d15e19eed54aefb6b96276fad98324 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:04.591821 11400 raft_consensus.cc:1261] T b4d15e19eed54aefb6b96276fad98324 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:04.593066 18261 consensus_queue.cc:1035] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:04.593427 18261 consensus_queue.cc:1035] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:04.596227 11112 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 5b7de08d2f004cd8abc2bbe1cf7b1444 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:00:04.596786 11121 catalog_manager.cc:5484] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 reported cstate change: config changed from index 6 to 7, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:04.596997 13185 raft_consensus.cc:2932] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:04.602042 16360 raft_consensus.cc:2932] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:04.605757 11123 catalog_manager.cc:5484] T b4d15e19eed54aefb6b96276fad98324 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 4 to 5, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:04.612744 13185 raft_consensus.cc:2932] T b4d15e19eed54aefb6b96276fad98324 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:04.615175 11248 tablet_service.cc:1514] Processing DeleteTablet for tablet b4d15e19eed54aefb6b96276fad98324 with delete_type TABLET_DATA_TOMBSTONED (TS e2dafc1866314669ad0de8fb7bf7fd05 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42432
I20240501 08:00:04.623724 19211 tablet_replica.cc:330] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05: stopping tablet replica
I20240501 08:00:04.623920 19211 raft_consensus.cc:2229] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:04.624176 19211 raft_consensus.cc:2258] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:04.624786 19211 ts_tablet_manager.cc:1875] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:04.627562 19211 ts_tablet_manager.cc:1888] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:00:04.628170 19211 log.cc:1196] T b4d15e19eed54aefb6b96276fad98324 P e2dafc1866314669ad0de8fb7bf7fd05: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b4d15e19eed54aefb6b96276fad98324
I20240501 08:00:04.629465 11113 catalog_manager.cc:4830] TS e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085): tablet b4d15e19eed54aefb6b96276fad98324 (table rebalance_test_table_8 [id=df5f9864bbf8499dbaa392bcb85701c0]) successfully deleted
I20240501 08:00:04.629907 11532 raft_consensus.cc:1261] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:00:04.630383 18263 consensus_queue.cc:1035] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:04.643059 11400 raft_consensus.cc:2932] T b4d15e19eed54aefb6b96276fad98324 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:04.646450 13185 raft_consensus.cc:1261] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:00:04.646951 18263 consensus_queue.cc:1035] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:04.654402 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.667037 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:04.676532 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.680210 11400 raft_consensus.cc:1261] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:00:04.681285 18991 consensus_queue.cc:1035] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:04.681883 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:04.704813 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.705544 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:04.711817 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.743762 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.748265 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:04.764374 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.782315 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.785959 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.803659 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:04.824924 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:04.836061 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.836707 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.852540 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.855659 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:04.858467 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.861093 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:04.867740 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.877645 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.884778 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:04.898686 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.902491 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.907016 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:04.921039 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:04.948544 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.954085 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.959561 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.963375 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:04.966984 11248 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:04.978327 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.979192 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:04.986564 13429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.988081 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:04.988720 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:04.990155 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:05.017345 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:05.020208 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:05.034320 19381 ts_tablet_manager.cc:902] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608: Initiating tablet copy from peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 08:00:05.034904 19381 tablet_copy_client.cc:308] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:44403
I20240501 08:00:05.035465 11683 tablet_copy_service.cc:140] P 603919bb02364c04b6c530245b8b02f2: Received BeginTabletCopySession request for tablet 5b7de08d2f004cd8abc2bbe1cf7b1444 from peer ca177138f2c44d2eafa1646b795f7608 ({username='slave'} at 127.0.99.136:46561)
I20240501 08:00:05.035604 11683 tablet_copy_service.cc:161] P 603919bb02364c04b6c530245b8b02f2: Beginning new tablet copy session on tablet 5b7de08d2f004cd8abc2bbe1cf7b1444 from peer ca177138f2c44d2eafa1646b795f7608 at {username='slave'} at 127.0.99.136:46561: session id = ca177138f2c44d2eafa1646b795f7608-5b7de08d2f004cd8abc2bbe1cf7b1444
I20240501 08:00:05.036574 11683 tablet_copy_source_session.cc:205] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:05.043627 19381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7de08d2f004cd8abc2bbe1cf7b1444. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:05.047612 19381 tablet_copy_client.cc:783] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:05.047804 19381 tablet_copy_client.cc:647] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:05.052903 19381 tablet_copy_client.cc:523] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:05.054618 19381 tablet_bootstrap.cc:492] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608: Bootstrap starting.
I20240501 08:00:05.087167 11269 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2e63d24d7df4fa3b2b26695e70ebe9f"
dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43842
I20240501 08:00:05.087316 11269 raft_consensus.cc:595] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:05.121209 11532 consensus_queue.cc:237] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:05.122701 11662 raft_consensus.cc:1261] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:00:05.123019 13448 raft_consensus.cc:1261] T a24be92aec7544089c7818f2802b9c85 P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:00:05.123071 19203 consensus_queue.cc:1035] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:05.123988 13185 raft_consensus.cc:1261] T a24be92aec7544089c7818f2802b9c85 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:00:05.124349 19203 consensus_queue.cc:1035] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:05.124521 16360 consensus_queue.cc:1035] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:05.127043 11662 consensus_queue.cc:237] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:05.128053 19203 raft_consensus.cc:2932] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:05.130277 13186 raft_consensus.cc:2932] T a24be92aec7544089c7818f2802b9c85 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:05.130972 11123 catalog_manager.cc:5484] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 5 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:05.132026 11269 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2e63d24d7df4fa3b2b26695e70ebe9f"
dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43864
I20240501 08:00:05.132153 11269 raft_consensus.cc:595] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:05.132215 13446 raft_consensus.cc:2932] T a24be92aec7544089c7818f2802b9c85 P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:05.133112 11662 raft_consensus.cc:2932] T a24be92aec7544089c7818f2802b9c85 P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:05.134135 13185 raft_consensus.cc:1261] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:05.134797 19454 consensus_queue.cc:1035] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:05.137140 19178 raft_consensus.cc:2932] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:05.141685 13185 raft_consensus.cc:2932] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:05.148386 19381 tablet_bootstrap.cc:492] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:05.148701 19381 tablet_bootstrap.cc:492] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608: Bootstrap complete.
I20240501 08:00:05.148977 19381 ts_tablet_manager.cc:1367] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608: Time spent bootstrapping tablet: real 0.094s user 0.025s sys 0.001s
I20240501 08:00:05.149451 19381 raft_consensus.cc:348] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:05.149564 19381 raft_consensus.cc:729] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca177138f2c44d2eafa1646b795f7608, State: Initialized, Role: FOLLOWER
I20240501 08:00:05.149745 19381 consensus_queue.cc:260] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:05.150117 19381 ts_tablet_manager.cc:1398] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:05.150627 11683 tablet_copy_service.cc:342] P 603919bb02364c04b6c530245b8b02f2: Request end of tablet copy session ca177138f2c44d2eafa1646b795f7608-5b7de08d2f004cd8abc2bbe1cf7b1444 received from {username='slave'} at 127.0.99.136:46561
I20240501 08:00:05.150728 11683 tablet_copy_service.cc:433] P 603919bb02364c04b6c530245b8b02f2: ending tablet copy session ca177138f2c44d2eafa1646b795f7608-5b7de08d2f004cd8abc2bbe1cf7b1444 on tablet 5b7de08d2f004cd8abc2bbe1cf7b1444 with peer ca177138f2c44d2eafa1646b795f7608
I20240501 08:00:05.154125 11248 tablet_service.cc:1514] Processing DeleteTablet for tablet a24be92aec7544089c7818f2802b9c85 with delete_type TABLET_DATA_TOMBSTONED (TS e2dafc1866314669ad0de8fb7bf7fd05 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42432
I20240501 08:00:05.155385 19480 tablet_replica.cc:330] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05: stopping tablet replica
I20240501 08:00:05.155537 19480 raft_consensus.cc:2229] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:05.155637 19480 raft_consensus.cc:2258] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:05.156311 11120 catalog_manager.cc:5484] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9 reported cstate change: config changed from index 5 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:05.156443 19480 ts_tablet_manager.cc:1875] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:05.159616 19480 ts_tablet_manager.cc:1888] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:00:05.159762 19480 log.cc:1196] T a24be92aec7544089c7818f2802b9c85 P e2dafc1866314669ad0de8fb7bf7fd05: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a24be92aec7544089c7818f2802b9c85
I20240501 08:00:05.160399 11113 catalog_manager.cc:4830] TS e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085): tablet a24be92aec7544089c7818f2802b9c85 (table rebalance_test_table_0 [id=24b4f6e8cc8c44f6961591cac7beacb4]) successfully deleted
I20240501 08:00:05.160766 11510 tablet_service.cc:1514] Processing DeleteTablet for tablet f80db2e37bbc4716ad3a5200c61ee97b with delete_type TABLET_DATA_TOMBSTONED (TS 22ec12774a4047a8b859743dc73f8ff6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52810
I20240501 08:00:05.161271 11662 consensus_queue.cc:237] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
W20240501 08:00:05.163607 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f80db2e37bbc4716ad3a5200c61ee97b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:05.168007 13185 raft_consensus.cc:1261] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:05.168521 18991 raft_consensus.cc:979] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05: : Instructing follower ca177138f2c44d2eafa1646b795f7608 to start an election
I20240501 08:00:05.168619 18991 raft_consensus.cc:1067] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Signalling peer ca177138f2c44d2eafa1646b795f7608 to start an election
I20240501 08:00:05.168695 19178 consensus_queue.cc:1035] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:05.169127 13446 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f2e63d24d7df4fa3b2b26695e70ebe9f"
dest_uuid: "ca177138f2c44d2eafa1646b795f7608"
from {username='slave'} at 127.0.99.129:36501
I20240501 08:00:05.169328 13446 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:05.169411 13446 raft_consensus.cc:3037] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:05.169894 19487 tablet_replica.cc:330] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6: stopping tablet replica
I20240501 08:00:05.170604 19487 raft_consensus.cc:2229] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:05.170778 19487 raft_consensus.cc:2258] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:05.170531 13446 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:05.172655 19454 raft_consensus.cc:2932] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:05.173089 11269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2e63d24d7df4fa3b2b26695e70ebe9f" candidate_uuid: "ca177138f2c44d2eafa1646b795f7608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 08:00:05.173211 11269 raft_consensus.cc:3032] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:05.173254 11269 raft_consensus.cc:729] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Running, Role: LEADER
I20240501 08:00:05.173223 13185 raft_consensus.cc:2932] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:05.173403 11269 consensus_queue.cc:260] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:05.173574 11269 raft_consensus.cc:3037] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:05.174533 11112 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f80db2e37bbc4716ad3a5200c61ee97b with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:00:05.174811 11662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2e63d24d7df4fa3b2b26695e70ebe9f" candidate_uuid: "ca177138f2c44d2eafa1646b795f7608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "603919bb02364c04b6c530245b8b02f2"
I20240501 08:00:05.174933 11662 raft_consensus.cc:3037] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:05.175542 11269 raft_consensus.cc:2454] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca177138f2c44d2eafa1646b795f7608 in term 2.
I20240501 08:00:05.175848 11662 raft_consensus.cc:2454] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca177138f2c44d2eafa1646b795f7608 in term 2.
I20240501 08:00:05.175817 11119 catalog_manager.cc:5484] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9 reported cstate change: config changed from index 6 to 7, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:05.176115 13446 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [CANDIDATE]: Term 2 election: Requested vote from peers 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403), e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 08:00:05.176522 11400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2e63d24d7df4fa3b2b26695e70ebe9f" candidate_uuid: "ca177138f2c44d2eafa1646b795f7608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 08:00:05.176576 13382 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, ca177138f2c44d2eafa1646b795f7608, e2dafc1866314669ad0de8fb7bf7fd05; no voters:
I20240501 08:00:05.176645 11400 raft_consensus.cc:3037] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:05.176821 19449 raft_consensus.cc:2781] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:05.177004 19449 raft_consensus.cc:686] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [term 2 LEADER]: Becoming Leader. State: Replica: ca177138f2c44d2eafa1646b795f7608, State: Running, Role: LEADER
I20240501 08:00:05.177320 19449 consensus_queue.cc:237] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:05.177685 11400 raft_consensus.cc:2454] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca177138f2c44d2eafa1646b795f7608 in term 2.
I20240501 08:00:05.180305 11121 catalog_manager.cc:5484] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 reported cstate change: term changed from 1 to 2, leader changed from e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) to ca177138f2c44d2eafa1646b795f7608 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "ca177138f2c44d2eafa1646b795f7608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:05.180871 19487 ts_tablet_manager.cc:1875] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:05.184903 19487 ts_tablet_manager.cc:1888] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:05.185678 19487 log.cc:1196] T f80db2e37bbc4716ad3a5200c61ee97b P 22ec12774a4047a8b859743dc73f8ff6: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/f80db2e37bbc4716ad3a5200c61ee97b
I20240501 08:00:05.186478 11114 catalog_manager.cc:4830] TS 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957): tablet f80db2e37bbc4716ad3a5200c61ee97b (table rebalance_test_table_4 [id=384b2b245c6b4931a4fc02c6b5d647b2]) successfully deleted
I20240501 08:00:05.252032 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:05.253558 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.263674 13429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:05.267544 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:05.272980 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.286306 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.298681 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.302004 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:05.307082 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:05.309397 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.314616 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:05.324558 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:05.335438 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:05.347419 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:05.363297 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.404525 13429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:05.433475 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.437839 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:05.451669 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.453886 13429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:05.457015 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.458137 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:05.482710 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:05.489286 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.497526 13429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:05.502764 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.519227 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.520287 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:05.524678 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:05.526635 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:05.526731 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:05.555397 13446 raft_consensus.cc:1203] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:00:05.567869 11401 raft_consensus.cc:1261] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer ca177138f2c44d2eafa1646b795f7608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:00:05.568428 19503 consensus_queue.cc:1035] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:05.575002 11269 raft_consensus.cc:1261] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Refusing update from remote peer ca177138f2c44d2eafa1646b795f7608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:00:05.576465 19503 consensus_queue.cc:1035] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:05.583698 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:05.596940 11662 raft_consensus.cc:1261] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Refusing update from remote peer ca177138f2c44d2eafa1646b795f7608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:00:05.597754 19449 consensus_queue.cc:1035] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:05.654471 11532 consensus_queue.cc:237] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:05.656160 13185 raft_consensus.cc:1261] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:05.657027 19206 consensus_queue.cc:1035] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:05.672026 19206 raft_consensus.cc:2932] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:05.676532 13185 raft_consensus.cc:2932] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:05.679656 11119 catalog_manager.cc:5484] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 5 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:05.683584 11642 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b0ff0769dc426190f038811155c741 with delete_type TABLET_DATA_TOMBSTONED (TS 603919bb02364c04b6c530245b8b02f2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53514
I20240501 08:00:05.684398 11532 consensus_queue.cc:237] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
W20240501 08:00:05.687247 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48b0ff0769dc426190f038811155c741. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:05.687727 13186 raft_consensus.cc:1261] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:05.689465 19206 consensus_queue.cc:1035] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:05.693163 19804 ts_tablet_manager.cc:902] T f80db2e37bbc4716ad3a5200c61ee97b P ca177138f2c44d2eafa1646b795f7608: Initiating tablet copy from peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 08:00:05.693212 11532 consensus_queue.cc:237] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:05.693766 19804 tablet_copy_client.cc:308] T f80db2e37bbc4716ad3a5200c61ee97b P ca177138f2c44d2eafa1646b795f7608: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:44403
I20240501 08:00:05.694259 11683 tablet_copy_service.cc:140] P 603919bb02364c04b6c530245b8b02f2: Received BeginTabletCopySession request for tablet f80db2e37bbc4716ad3a5200c61ee97b from peer ca177138f2c44d2eafa1646b795f7608 ({username='slave'} at 127.0.99.136:46561)
I20240501 08:00:05.694396 11683 tablet_copy_service.cc:161] P 603919bb02364c04b6c530245b8b02f2: Beginning new tablet copy session on tablet f80db2e37bbc4716ad3a5200c61ee97b from peer ca177138f2c44d2eafa1646b795f7608 at {username='slave'} at 127.0.99.136:46561: session id = ca177138f2c44d2eafa1646b795f7608-f80db2e37bbc4716ad3a5200c61ee97b
I20240501 08:00:05.694658 13446 raft_consensus.cc:1261] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:00:05.695201 19206 consensus_queue.cc:1035] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:05.695365 11683 tablet_copy_source_session.cc:205] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:05.697770 19804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80db2e37bbc4716ad3a5200c61ee97b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:05.700107 19802 raft_consensus.cc:2932] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:05.700850 13446 raft_consensus.cc:2932] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:05.701795 19804 tablet_copy_client.cc:783] T f80db2e37bbc4716ad3a5200c61ee97b P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:05.701928 19804 tablet_copy_client.cc:647] T f80db2e37bbc4716ad3a5200c61ee97b P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:05.703678 11123 catalog_manager.cc:5484] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 reported cstate change: config changed from index 6 to 7, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:05.706744 11380 tablet_service.cc:1514] Processing DeleteTablet for tablet 90f11b5693d7456d8a604e6c8726e69a with delete_type TABLET_DATA_TOMBSTONED (TS 655ebda1fb2c49c4af7087a760529835 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:43354
I20240501 08:00:05.708587 11532 consensus_queue.cc:237] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
W20240501 08:00:05.712632 11465 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): Couldn't send request to peer 603919bb02364c04b6c530245b8b02f2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:05.712925 19817 tablet_replica.cc:330] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835: stopping tablet replica
I20240501 08:00:05.713091 19817 raft_consensus.cc:2229] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:05.713255 19817 raft_consensus.cc:2258] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:05.713241 13446 raft_consensus.cc:1261] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:00:05.713624 19816 consensus_queue.cc:1035] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:05.713956 19817 ts_tablet_manager.cc:1875] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:05.716012 19802 raft_consensus.cc:2932] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:00:05.716619 19817 ts_tablet_manager.cc:1888] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:00:05.717842 19817 log.cc:1196] T 90f11b5693d7456d8a604e6c8726e69a P 655ebda1fb2c49c4af7087a760529835: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/90f11b5693d7456d8a604e6c8726e69a
I20240501 08:00:05.717759 13446 raft_consensus.cc:2932] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:00:05.717818 19206 raft_consensus.cc:2932] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:05.718544 11112 catalog_manager.cc:4830] TS 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575): tablet 90f11b5693d7456d8a604e6c8726e69a (table rebalance_test_table_5 [id=0af21d4c4fcb49a7ac0756a5ded5d57d]) successfully deleted
I20240501 08:00:05.718912 19804 tablet_copy_client.cc:523] T f80db2e37bbc4716ad3a5200c61ee97b P ca177138f2c44d2eafa1646b795f7608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:05.720288 19804 tablet_bootstrap.cc:492] T f80db2e37bbc4716ad3a5200c61ee97b P ca177138f2c44d2eafa1646b795f7608: Bootstrap starting.
I20240501 08:00:05.720546 13186 raft_consensus.cc:2932] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:05.720667 19801 tablet_replica.cc:330] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: stopping tablet replica
I20240501 08:00:05.720822 19801 raft_consensus.cc:2229] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:05.720988 19801 raft_consensus.cc:2258] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:05.721588 19801 ts_tablet_manager.cc:1875] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:05.722466 11114 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 48b0ff0769dc426190f038811155c741 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:00:05.723482 11123 catalog_manager.cc:5484] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9 reported cstate change: config changed from index 6 to 7, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:05.723723 11114 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 90f11b5693d7456d8a604e6c8726e69a with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:00:05.725407 11121 catalog_manager.cc:5484] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 reported cstate change: config changed from index 7 to 8, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:00:05.736941 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:05.738212 19801 ts_tablet_manager.cc:1888] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:00:05.738379 19801 log.cc:1196] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/48b0ff0769dc426190f038811155c741
I20240501 08:00:05.740089 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.752593 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:05.756820 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:05.744747 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:05.764645 11112 catalog_manager.cc:4830] TS 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): tablet 48b0ff0769dc426190f038811155c741 (table rebalance_test_table_3 [id=6447707eb34f45d7a8d3704cd09d7786]) successfully deleted
I20240501 08:00:05.764762 13429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:05.772679 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.778249 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:05.779884 13429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:05.781206 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.781953 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:05.812628 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:05.821586 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:05.824419 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.827591 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.836056 19804 tablet_bootstrap.cc:492] T f80db2e37bbc4716ad3a5200c61ee97b P ca177138f2c44d2eafa1646b795f7608: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:05.836414 19804 tablet_bootstrap.cc:492] T f80db2e37bbc4716ad3a5200c61ee97b P ca177138f2c44d2eafa1646b795f7608: Bootstrap complete.
I20240501 08:00:05.836673 19804 ts_tablet_manager.cc:1367] T f80db2e37bbc4716ad3a5200c61ee97b P ca177138f2c44d2eafa1646b795f7608: Time spent bootstrapping tablet: real 0.116s user 0.025s sys 0.003s
I20240501 08:00:05.837152 19804 raft_consensus.cc:348] T f80db2e37bbc4716ad3a5200c61ee97b P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:05.837268 19804 raft_consensus.cc:729] T f80db2e37bbc4716ad3a5200c61ee97b P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca177138f2c44d2eafa1646b795f7608, State: Initialized, Role: FOLLOWER
I20240501 08:00:05.837409 19804 consensus_queue.cc:260] T f80db2e37bbc4716ad3a5200c61ee97b P ca177138f2c44d2eafa1646b795f7608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:05.837795 19804 ts_tablet_manager.cc:1398] T f80db2e37bbc4716ad3a5200c61ee97b P ca177138f2c44d2eafa1646b795f7608: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:00:05.838229 11683 tablet_copy_service.cc:342] P 603919bb02364c04b6c530245b8b02f2: Request end of tablet copy session ca177138f2c44d2eafa1646b795f7608-f80db2e37bbc4716ad3a5200c61ee97b received from {username='slave'} at 127.0.99.136:46561
I20240501 08:00:05.838303 11683 tablet_copy_service.cc:433] P 603919bb02364c04b6c530245b8b02f2: ending tablet copy session ca177138f2c44d2eafa1646b795f7608-f80db2e37bbc4716ad3a5200c61ee97b on tablet f80db2e37bbc4716ad3a5200c61ee97b with peer ca177138f2c44d2eafa1646b795f7608
I20240501 08:00:05.875532 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.937805 11401 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e8550ea95984d08937b9b3e2af12bec"
dest_uuid: "655ebda1fb2c49c4af7087a760529835"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52670
I20240501 08:00:05.937987 11401 raft_consensus.cc:595] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:05.939292 11269 consensus_queue.cc:237] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:05.941408 11532 raft_consensus.cc:1261] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:05.942348 13446 raft_consensus.cc:1261] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:05.943116 18991 consensus_queue.cc:1035] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:05.945362 18991 consensus_queue.cc:1035] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:05.950953 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:05.960731 11510 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.961402 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:05.961772 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.962963 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:05.963312 11380 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:05.963486 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:05.972635 18991 raft_consensus.cc:2932] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:05.975463 13446 raft_consensus.cc:2932] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:05.975816 11532 raft_consensus.cc:2932] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:05.978485 11121 catalog_manager.cc:5484] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index 4 to 6, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:05.981765 11380 tablet_service.cc:1514] Processing DeleteTablet for tablet b9431cf5853c4148ac38855b1c083d4f with delete_type TABLET_DATA_TOMBSTONED (TS 655ebda1fb2c49c4af7087a760529835 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43354
I20240501 08:00:05.982008 19817 tablet_replica.cc:330] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835: stopping tablet replica
I20240501 08:00:05.982149 19817 raft_consensus.cc:2229] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:05.982267 19817 raft_consensus.cc:2258] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:05.982925 19817 ts_tablet_manager.cc:1875] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:05.985095 19817 ts_tablet_manager.cc:1888] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:00:05.985152 19817 log.cc:1196] T b9431cf5853c4148ac38855b1c083d4f P 655ebda1fb2c49c4af7087a760529835: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b9431cf5853c4148ac38855b1c083d4f
I20240501 08:00:05.985673 11112 catalog_manager.cc:4830] TS 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575): tablet b9431cf5853c4148ac38855b1c083d4f (table rebalance_test_table_1 [id=98fb02bdb5f54b1699749e3c8d588a5e]) successfully deleted
I20240501 08:00:05.992341 13429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:06.096669 18761 raft_consensus.cc:979] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: : Instructing follower 22ec12774a4047a8b859743dc73f8ff6 to start an election
I20240501 08:00:06.096765 18761 raft_consensus.cc:1067] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Signalling peer 22ec12774a4047a8b859743dc73f8ff6 to start an election
I20240501 08:00:06.097441 11532 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e8550ea95984d08937b9b3e2af12bec"
dest_uuid: "22ec12774a4047a8b859743dc73f8ff6"
from {username='slave'} at 127.0.99.130:44689
I20240501 08:00:06.097596 11532 raft_consensus.cc:482] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:06.097651 11532 raft_consensus.cc:3037] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:06.099520 11532 raft_consensus.cc:504] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:06.099903 11532 leader_election.cc:288] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 2 election: Requested vote from peers 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
I20240501 08:00:06.101137 13053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8550ea95984d08937b9b3e2af12bec" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "60e8857abe7c41038af46ebb5a095a90"
I20240501 08:00:06.101270 13053 raft_consensus.cc:3037] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:06.102221 13053 raft_consensus.cc:2454] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 2.
I20240501 08:00:06.102638 11467 leader_election.cc:302] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ec12774a4047a8b859743dc73f8ff6, 60e8857abe7c41038af46ebb5a095a90; no voters:
I20240501 08:00:06.102829 19840 raft_consensus.cc:2781] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:06.102911 19840 raft_consensus.cc:686] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Becoming Leader. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Running, Role: LEADER
I20240501 08:00:06.103068 19840 consensus_queue.cc:237] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:06.106083 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8550ea95984d08937b9b3e2af12bec" candidate_uuid: "22ec12774a4047a8b859743dc73f8ff6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 08:00:06.106217 11401 raft_consensus.cc:3032] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:06.106276 11401 raft_consensus.cc:729] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Running, Role: LEADER
I20240501 08:00:06.106412 11401 consensus_queue.cc:260] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:06.106626 11401 raft_consensus.cc:3037] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:06.107761 11401 raft_consensus.cc:2454] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ec12774a4047a8b859743dc73f8ff6 in term 2.
I20240501 08:00:06.107882 11119 catalog_manager.cc:5484] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: term changed from 1 to 2, leader changed from 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) to 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:06.115096 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:06.118688 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:06.124750 13429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:06.137773 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:06.146950 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:06.151531 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:06.157909 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:06.166577 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:06.186698 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:06.187805 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:06.196679 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:06.200176 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:06.222954 20094 ts_tablet_manager.cc:902] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: Initiating tablet copy from peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 08:00:06.223004 13446 raft_consensus.cc:1203] T f80db2e37bbc4716ad3a5200c61ee97b P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:00:06.223515 20090 ts_tablet_manager.cc:902] T 48b0ff0769dc426190f038811155c741 P e5c03261d31c4d9a91077a17e1cd0336: Initiating tablet copy from peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 08:00:06.224128 20094 tablet_copy_client.cc:308] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37957
I20240501 08:00:06.224756 11552 tablet_copy_service.cc:140] P 22ec12774a4047a8b859743dc73f8ff6: Received BeginTabletCopySession request for tablet 90f11b5693d7456d8a604e6c8726e69a from peer 603919bb02364c04b6c530245b8b02f2 ({username='slave'} at 127.0.99.132:54785)
I20240501 08:00:06.224867 11552 tablet_copy_service.cc:161] P 22ec12774a4047a8b859743dc73f8ff6: Beginning new tablet copy session on tablet 90f11b5693d7456d8a604e6c8726e69a from peer 603919bb02364c04b6c530245b8b02f2 at {username='slave'} at 127.0.99.132:54785: session id = 603919bb02364c04b6c530245b8b02f2-90f11b5693d7456d8a604e6c8726e69a
I20240501 08:00:06.225064 20090 tablet_copy_client.cc:308] T 48b0ff0769dc426190f038811155c741 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37957
I20240501 08:00:06.225801 11551 tablet_copy_service.cc:140] P 22ec12774a4047a8b859743dc73f8ff6: Received BeginTabletCopySession request for tablet 48b0ff0769dc426190f038811155c741 from peer e5c03261d31c4d9a91077a17e1cd0336 ({username='slave'} at 127.0.99.135:47673)
I20240501 08:00:06.225873 11552 tablet_copy_source_session.cc:205] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:06.225901 11551 tablet_copy_service.cc:161] P 22ec12774a4047a8b859743dc73f8ff6: Beginning new tablet copy session on tablet 48b0ff0769dc426190f038811155c741 from peer e5c03261d31c4d9a91077a17e1cd0336 at {username='slave'} at 127.0.99.135:47673: session id = e5c03261d31c4d9a91077a17e1cd0336-48b0ff0769dc426190f038811155c741
I20240501 08:00:06.226979 20094 ts_tablet_manager.cc:1875] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:00:06.227027 11551 tablet_copy_source_session.cc:205] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:06.228479 20090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b0ff0769dc426190f038811155c741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:06.232108 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:06.233366 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:06.236279 20094 ts_tablet_manager.cc:1888] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.5
I20240501 08:00:06.236508 20094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f11b5693d7456d8a604e6c8726e69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:06.237939 20090 tablet_copy_client.cc:783] T 48b0ff0769dc426190f038811155c741 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:06.238173 20090 tablet_copy_client.cc:647] T 48b0ff0769dc426190f038811155c741 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:06.238579 20094 tablet_copy_client.cc:783] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:06.238940 20094 tablet_copy_client.cc:647] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:06.239599 20090 tablet_copy_client.cc:523] T 48b0ff0769dc426190f038811155c741 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:06.240864 20090 tablet_bootstrap.cc:492] T 48b0ff0769dc426190f038811155c741 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap starting.
I20240501 08:00:06.243824 20094 tablet_copy_client.cc:523] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:06.246039 20094 tablet_bootstrap.cc:492] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 08:00:06.269094 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:06.271238 13429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:06.272442 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:06.278667 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:06.304759 20090 tablet_bootstrap.cc:492] T 48b0ff0769dc426190f038811155c741 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:06.305178 20090 tablet_bootstrap.cc:492] T 48b0ff0769dc426190f038811155c741 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap complete.
I20240501 08:00:06.305840 20090 ts_tablet_manager.cc:1367] T 48b0ff0769dc426190f038811155c741 P e5c03261d31c4d9a91077a17e1cd0336: Time spent bootstrapping tablet: real 0.065s user 0.027s sys 0.000s
I20240501 08:00:06.306392 20090 raft_consensus.cc:348] T 48b0ff0769dc426190f038811155c741 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:06.308079 20090 raft_consensus.cc:729] T 48b0ff0769dc426190f038811155c741 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c03261d31c4d9a91077a17e1cd0336, State: Initialized, Role: FOLLOWER
I20240501 08:00:06.308310 20090 consensus_queue.cc:260] T 48b0ff0769dc426190f038811155c741 P e5c03261d31c4d9a91077a17e1cd0336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:06.308997 20090 ts_tablet_manager.cc:1398] T 48b0ff0769dc426190f038811155c741 P e5c03261d31c4d9a91077a17e1cd0336: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 08:00:06.310809 11551 tablet_copy_service.cc:342] P 22ec12774a4047a8b859743dc73f8ff6: Request end of tablet copy session e5c03261d31c4d9a91077a17e1cd0336-48b0ff0769dc426190f038811155c741 received from {username='slave'} at 127.0.99.135:47673
I20240501 08:00:06.310905 11551 tablet_copy_service.cc:433] P 22ec12774a4047a8b859743dc73f8ff6: ending tablet copy session e5c03261d31c4d9a91077a17e1cd0336-48b0ff0769dc426190f038811155c741 on tablet 48b0ff0769dc426190f038811155c741 with peer e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:00:06.315624 13429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:06.321435 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:06.328897 13429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:06.332653 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:06.345103 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:06.345402 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:06.352823 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:06.358690 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:06.366065 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:06.371395 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:06.373174 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:06.374076 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:06.382122 13297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:06.382325 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:06.463156 11269 consensus_queue.cc:237] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:06.464664 13054 raft_consensus.cc:1261] T b21b6e2023e842739a7c0cc946e7c2fd P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:06.464793 11401 raft_consensus.cc:1261] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:06.465137 18991 consensus_queue.cc:1035] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:06.465528 18991 consensus_queue.cc:1035] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:06.471021 18953 raft_consensus.cc:2932] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:06.471761 11401 raft_consensus.cc:2932] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:06.473529 13054 raft_consensus.cc:2932] T b21b6e2023e842739a7c0cc946e7c2fd P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:06.475131 11119 catalog_manager.cc:5484] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 4 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:06.476624 20094 tablet_bootstrap.cc:492] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:06.476958 20094 tablet_bootstrap.cc:492] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: Bootstrap complete.
I20240501 08:00:06.477156 20094 ts_tablet_manager.cc:1367] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.231s user 0.030s sys 0.000s
I20240501 08:00:06.477661 20094 raft_consensus.cc:348] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:00:06.477773 20094 raft_consensus.cc:729] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 08:00:06.478206 11642 tablet_service.cc:1514] Processing DeleteTablet for tablet b21b6e2023e842739a7c0cc946e7c2fd with delete_type TABLET_DATA_TOMBSTONED (TS 603919bb02364c04b6c530245b8b02f2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53514
I20240501 08:00:06.478300 20094 consensus_queue.cc:260] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:00:06.479139 20094 ts_tablet_manager.cc:1398] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:06.479665 11551 tablet_copy_service.cc:342] P 22ec12774a4047a8b859743dc73f8ff6: Request end of tablet copy session 603919bb02364c04b6c530245b8b02f2-90f11b5693d7456d8a604e6c8726e69a received from {username='slave'} at 127.0.99.132:54785
I20240501 08:00:06.479735 11551 tablet_copy_service.cc:433] P 22ec12774a4047a8b859743dc73f8ff6: ending tablet copy session 603919bb02364c04b6c530245b8b02f2-90f11b5693d7456d8a604e6c8726e69a on tablet 90f11b5693d7456d8a604e6c8726e69a with peer 603919bb02364c04b6c530245b8b02f2
I20240501 08:00:06.495128 13054 raft_consensus.cc:1261] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:06.495764 19816 consensus_queue.cc:1035] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:06.496946 11269 consensus_queue.cc:237] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:06.497674 20201 tablet_replica.cc:330] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: stopping tablet replica
I20240501 08:00:06.498863 13054 raft_consensus.cc:1261] T 7a266f266c3d4fd49fa8b325f1045d61 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:06.499128 11401 raft_consensus.cc:1261] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:00:06.499361 11400 raft_consensus.cc:1261] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:06.499989 19840 consensus_queue.cc:1035] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:06.500228 20201 raft_consensus.cc:2229] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:06.500447 20201 raft_consensus.cc:2258] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:06.500427 20198 consensus_queue.cc:1035] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:06.500772 20198 consensus_queue.cc:1035] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:00:06.501271 20201 ts_tablet_manager.cc:1875] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:06.508646 20201 ts_tablet_manager.cc:1888] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:00:06.508757 20201 log.cc:1196] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b21b6e2023e842739a7c0cc946e7c2fd
I20240501 08:00:06.509500 11112 catalog_manager.cc:4830] TS 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): tablet b21b6e2023e842739a7c0cc946e7c2fd (table rebalance_test_table_5 [id=0af21d4c4fcb49a7ac0756a5ded5d57d]) successfully deleted
I20240501 08:00:06.510327 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:06.519747 20198 raft_consensus.cc:2932] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:06.523844 11122 catalog_manager.cc:5484] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index 4 to 5, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:06.525168 11400 raft_consensus.cc:2932] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:06.527992 11642 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a266f266c3d4fd49fa8b325f1045d61 with delete_type TABLET_DATA_TOMBSTONED (TS 603919bb02364c04b6c530245b8b02f2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:53514
I20240501 08:00:06.528810 20201 tablet_replica.cc:330] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2: stopping tablet replica
I20240501 08:00:06.528980 20201 raft_consensus.cc:2229] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:06.529142 20201 raft_consensus.cc:2258] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:06.529803 20201 ts_tablet_manager.cc:1875] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:06.530992 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:06.535759 20201 ts_tablet_manager.cc:1888] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:00:06.536018 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:06.536645 20201 log.cc:1196] T 7a266f266c3d4fd49fa8b325f1045d61 P 603919bb02364c04b6c530245b8b02f2: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7a266f266c3d4fd49fa8b325f1045d61
I20240501 08:00:06.536652 13053 raft_consensus.cc:2932] T 7a266f266c3d4fd49fa8b325f1045d61 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:06.537277 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:06.547009 11112 catalog_manager.cc:4830] TS 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): tablet 7a266f266c3d4fd49fa8b325f1045d61 (table rebalance_test_table_4 [id=384b2b245c6b4931a4fc02c6b5d647b2]) successfully deleted
I20240501 08:00:06.551112 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:06.553089 13429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:06.557554 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:06.558001 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:06.565259 11661 raft_consensus.cc:1203] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 08:00:06.632102 11400 consensus_queue.cc:237] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:06.633785 13446 raft_consensus.cc:1261] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:06.634297 18777 consensus_queue.cc:1035] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:06.634573 11269 raft_consensus.cc:1261] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:06.635038 18777 consensus_queue.cc:1035] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:06.636579 18761 raft_consensus.cc:2932] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:06.640089 11122 catalog_manager.cc:5484] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 4 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:06.642635 13446 raft_consensus.cc:2932] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:06.644170 13448 consensus_queue.cc:237] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:06.648754 11269 raft_consensus.cc:2932] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:06.651325 11661 raft_consensus.cc:1261] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Refusing update from remote peer ca177138f2c44d2eafa1646b795f7608: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:06.651432 11400 raft_consensus.cc:1261] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer ca177138f2c44d2eafa1646b795f7608: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:06.651861 20329 consensus_queue.cc:1035] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:06.653295 20329 consensus_queue.cc:1035] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:06.656193 19503 raft_consensus.cc:2932] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:06.658506 11642 tablet_service.cc:1514] Processing DeleteTablet for tablet 53193305f868433c852755e8a8a81766 with delete_type TABLET_DATA_TOMBSTONED (TS 603919bb02364c04b6c530245b8b02f2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53514
I20240501 08:00:06.658875 20201 tablet_replica.cc:330] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2: stopping tablet replica
I20240501 08:00:06.659054 20201 raft_consensus.cc:2229] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:06.659214 20201 raft_consensus.cc:2258] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:06.659868 20201 ts_tablet_manager.cc:1875] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:06.665530 20201 ts_tablet_manager.cc:1888] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:00:06.665727 20201 log.cc:1196] T 53193305f868433c852755e8a8a81766 P 603919bb02364c04b6c530245b8b02f2: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/53193305f868433c852755e8a8a81766
I20240501 08:00:06.666292 11112 catalog_manager.cc:4830] TS 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): tablet 53193305f868433c852755e8a8a81766 (table rebalance_test_table_7 [id=36e20074d8dc4402bd7d9b2039213af5]) successfully deleted
I20240501 08:00:06.666211 11661 raft_consensus.cc:2932] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:06.666648 11401 raft_consensus.cc:2932] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:06.670094 11122 catalog_manager.cc:5484] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 4 to 6, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "ca177138f2c44d2eafa1646b795f7608" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:06.675261 11248 tablet_service.cc:1514] Processing DeleteTablet for tablet f2e63d24d7df4fa3b2b26695e70ebe9f with delete_type TABLET_DATA_TOMBSTONED (TS e2dafc1866314669ad0de8fb7bf7fd05 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42432
I20240501 08:00:06.680908 20375 tablet_replica.cc:330] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05: stopping tablet replica
I20240501 08:00:06.681167 20375 raft_consensus.cc:2229] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:06.681354 20375 raft_consensus.cc:2258] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:06.681931 20375 ts_tablet_manager.cc:1875] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:06.684589 20375 ts_tablet_manager.cc:1888] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:00:06.684749 20375 log.cc:1196] T f2e63d24d7df4fa3b2b26695e70ebe9f P e2dafc1866314669ad0de8fb7bf7fd05: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f2e63d24d7df4fa3b2b26695e70ebe9f
I20240501 08:00:06.685428 11113 catalog_manager.cc:4830] TS e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085): tablet f2e63d24d7df4fa3b2b26695e70ebe9f (table rebalance_test_table_2 [id=69f109081c7449be8819974c5363fa32]) successfully deleted
I20240501 08:00:06.770066 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:06.770838 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:06.778124 13316 raft_consensus.cc:1203] T 48b0ff0769dc426190f038811155c741 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:00:06.783126 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:06.783466 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:06.797919 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:06.810104 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:06.812884 13429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:06.813179 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:06.815917 13429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:06.825155 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:06.838755 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:06.864681 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:06.875609 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:06.894871 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:06.906280 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:06.915036 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:06.937045 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:06.945295 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:06.950950 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:06.953775 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:06.958958 13429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:06.965999 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:07.003077 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:07.012115 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:07.041786 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:07.043099 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:07.066545 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:07.070230 13429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:07.071034 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:07.080206 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:07.084157 11268 consensus_queue.cc:237] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:07.084800 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:07.085980 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:07.086169 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:07.087852 13186 raft_consensus.cc:1261] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:07.089033 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:07.089124 11401 raft_consensus.cc:1261] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:07.089426 20199 consensus_queue.cc:1035] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:07.089948 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:07.096789 20199 consensus_queue.cc:1035] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:07.101617 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:07.115061 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:07.120565 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:07.147271 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:07.154969 20199 raft_consensus.cc:2932] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:07.157512 11401 raft_consensus.cc:2932] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:07.162770 13429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:07.166368 11127 catalog_manager.cc:5484] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index 4 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:07.172106 11510 tablet_service.cc:1514] Processing DeleteTablet for tablet 17bb1ff829bc47a6a0bd6030517652aa with delete_type TABLET_DATA_TOMBSTONED (TS 22ec12774a4047a8b859743dc73f8ff6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52810
I20240501 08:00:07.173036 13186 raft_consensus.cc:2932] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:07.175168 11401 consensus_queue.cc:237] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:07.176683 13054 raft_consensus.cc:1261] T 164fd265561e47128e7ff585b8cfab07 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:07.177116 18777 consensus_queue.cc:1035] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:07.178699 13446 raft_consensus.cc:1261] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:07.179349 18777 consensus_queue.cc:1035] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:07.187835 20615 tablet_replica.cc:330] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6: stopping tablet replica
I20240501 08:00:07.187683 18777 raft_consensus.cc:2932] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:07.187996 20615 raft_consensus.cc:2229] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:07.188122 20615 raft_consensus.cc:2258] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:07.188666 20615 ts_tablet_manager.cc:1875] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:07.191956 20615 ts_tablet_manager.cc:1888] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:00:07.192030 20615 log.cc:1196] T 17bb1ff829bc47a6a0bd6030517652aa P 22ec12774a4047a8b859743dc73f8ff6: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/17bb1ff829bc47a6a0bd6030517652aa
I20240501 08:00:07.191865 13054 raft_consensus.cc:2932] T 164fd265561e47128e7ff585b8cfab07 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:07.192543 11114 catalog_manager.cc:4830] TS 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957): tablet 17bb1ff829bc47a6a0bd6030517652aa (table rebalance_test_table_3 [id=6447707eb34f45d7a8d3704cd09d7786]) successfully deleted
I20240501 08:00:07.193689 13446 raft_consensus.cc:2932] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:07.196180 11127 catalog_manager.cc:5484] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 6 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:07.200337 11401 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53193305f868433c852755e8a8a81766"
dest_uuid: "655ebda1fb2c49c4af7087a760529835"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52902
I20240501 08:00:07.200489 11401 raft_consensus.cc:595] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 2 LEADER]: Received request to transfer leadership
I20240501 08:00:07.200950 11248 tablet_service.cc:1514] Processing DeleteTablet for tablet 164fd265561e47128e7ff585b8cfab07 with delete_type TABLET_DATA_TOMBSTONED (TS e2dafc1866314669ad0de8fb7bf7fd05 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42432
I20240501 08:00:07.205116 20627 tablet_replica.cc:330] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05: stopping tablet replica
I20240501 08:00:07.205356 20627 raft_consensus.cc:2229] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:07.205588 20627 raft_consensus.cc:2258] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:07.206169 20627 ts_tablet_manager.cc:1875] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:07.209092 20627 ts_tablet_manager.cc:1888] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:00:07.209178 20627 log.cc:1196] T 164fd265561e47128e7ff585b8cfab07 P e2dafc1866314669ad0de8fb7bf7fd05: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/164fd265561e47128e7ff585b8cfab07
I20240501 08:00:07.209722 11113 catalog_manager.cc:4830] TS e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085): tablet 164fd265561e47128e7ff585b8cfab07 (table rebalance_test_table_6 [id=701e50fb54354219a26221f76a7352d2]) successfully deleted
I20240501 08:00:07.246446 11532 consensus_queue.cc:237] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:07.247156 13186 consensus_queue.cc:237] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:07.247773 13446 raft_consensus.cc:1261] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:00:07.248545 19840 consensus_queue.cc:1035] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:07.249219 11661 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Refusing update from remote peer 53d1d2c0767b4f6195b69758a3dd6cf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:00:07.249356 13054 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Refusing update from remote peer 53d1d2c0767b4f6195b69758a3dd6cf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:00:07.249712 11401 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer 53d1d2c0767b4f6195b69758a3dd6cf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:00:07.249778 18943 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:07.250656 20707 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:07.252136 20708 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:07.252851 18943 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:07.253899 11662 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:07.255075 19840 raft_consensus.cc:2932] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:07.255537 13446 raft_consensus.cc:2932] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:07.255688 13054 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:07.257913 11127 catalog_manager.cc:5484] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 reported cstate change: config changed from index 5 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:07.257956 11401 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:07.260079 11126 catalog_manager.cc:5484] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 8 to 9, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:07.262526 11642 tablet_service.cc:1514] Processing DeleteTablet for tablet 90f11b5693d7456d8a604e6c8726e69a with delete_type TABLET_DATA_TOMBSTONED (TS 603919bb02364c04b6c530245b8b02f2 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53514
I20240501 08:00:07.262903 20725 tablet_replica.cc:330] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: stopping tablet replica
I20240501 08:00:07.263052 20725 raft_consensus.cc:2229] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:07.263171 20725 raft_consensus.cc:2258] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:07.263744 20725 ts_tablet_manager.cc:1875] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:07.264117 11532 consensus_queue.cc:237] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:07.266636 11248 tablet_service.cc:1514] Processing DeleteTablet for tablet 25cee7e51c7c438aa721d71fb82dc71e with delete_type TABLET_DATA_TOMBSTONED (TS e2dafc1866314669ad0de8fb7bf7fd05 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42432
I20240501 08:00:07.266878 20627 tablet_replica.cc:330] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: stopping tablet replica
I20240501 08:00:07.267024 20627 raft_consensus.cc:2229] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:07.267153 20627 raft_consensus.cc:2258] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240501 08:00:07.267593 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90f11b5693d7456d8a604e6c8726e69a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:07.267812 20627 ts_tablet_manager.cc:1875] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:07.268302 13446 raft_consensus.cc:1261] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:00:07.268692 19840 consensus_queue.cc:1035] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:07.270196 20627 ts_tablet_manager.cc:1888] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:00:07.271039 20627 log.cc:1196] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/25cee7e51c7c438aa721d71fb82dc71e
I20240501 08:00:07.271327 20732 raft_consensus.cc:2932] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:07.271733 20725 ts_tablet_manager.cc:1888] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 08:00:07.271806 20725 log.cc:1196] T 90f11b5693d7456d8a604e6c8726e69a P 603919bb02364c04b6c530245b8b02f2: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/90f11b5693d7456d8a604e6c8726e69a
I20240501 08:00:07.271800 11113 catalog_manager.cc:4830] TS e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085): tablet 25cee7e51c7c438aa721d71fb82dc71e (table rebalance_test_table_2 [id=69f109081c7449be8819974c5363fa32]) successfully deleted
I20240501 08:00:07.271745 13446 raft_consensus.cc:2932] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:07.272293 11112 catalog_manager.cc:4830] TS 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): tablet 90f11b5693d7456d8a604e6c8726e69a (table rebalance_test_table_5 [id=0af21d4c4fcb49a7ac0756a5ded5d57d]) successfully deleted
I20240501 08:00:07.274853 11114 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 90f11b5693d7456d8a604e6c8726e69a with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:00:07.274772 11121 catalog_manager.cc:5484] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 reported cstate change: config changed from index 9 to 10, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:07.330979 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:07.336760 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:07.339622 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:07.340066 13429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:07.365080 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:07.369915 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:07.378612 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:07.392395 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:07.457516 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:07.460958 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:07.465003 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:07.467304 11380 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:07.470616 13429 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:07.476189 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:07.497541 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:07.498176 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:07.559419 20621 raft_consensus.cc:979] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835: : Instructing follower ca177138f2c44d2eafa1646b795f7608 to start an election
I20240501 08:00:07.559553 20621 raft_consensus.cc:1067] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 2 LEADER]: Signalling peer ca177138f2c44d2eafa1646b795f7608 to start an election
I20240501 08:00:07.560298 13446 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53193305f868433c852755e8a8a81766"
dest_uuid: "ca177138f2c44d2eafa1646b795f7608"
from {username='slave'} at 127.0.99.130:51565
I20240501 08:00:07.560451 13446 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:07.560515 13446 raft_consensus.cc:3037] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:07.561621 13446 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:07.562031 13446 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [CANDIDATE]: Term 3 election: Requested vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 08:00:07.562682 11268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53193305f868433c852755e8a8a81766" candidate_uuid: "ca177138f2c44d2eafa1646b795f7608" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 08:00:07.562847 11268 raft_consensus.cc:3037] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:07.563994 11268 raft_consensus.cc:2454] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca177138f2c44d2eafa1646b795f7608 in term 3.
I20240501 08:00:07.566552 11401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53193305f868433c852755e8a8a81766" candidate_uuid: "ca177138f2c44d2eafa1646b795f7608" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "655ebda1fb2c49c4af7087a760529835"
I20240501 08:00:07.566699 11401 raft_consensus.cc:3032] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:00:07.566748 11401 raft_consensus.cc:729] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Running, Role: LEADER
I20240501 08:00:07.566885 11401 consensus_queue.cc:260] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:07.567003 11401 raft_consensus.cc:3037] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:00:07.567299 11268 raft_consensus.cc:1226] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 3 FOLLOWER]: Rejecting Update request from peer 655ebda1fb2c49c4af7087a760529835 for earlier term 2. Current term is 3. Ops: []
I20240501 08:00:07.568040 11401 raft_consensus.cc:2454] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca177138f2c44d2eafa1646b795f7608 in term 3.
I20240501 08:00:07.568395 13382 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835, ca177138f2c44d2eafa1646b795f7608; no voters:
I20240501 08:00:07.568789 20329 raft_consensus.cc:2781] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:00:07.568965 20329 raft_consensus.cc:686] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [term 3 LEADER]: Becoming Leader. State: Replica: ca177138f2c44d2eafa1646b795f7608, State: Running, Role: LEADER
I20240501 08:00:07.569120 20329 consensus_queue.cc:237] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:07.571264 11120 catalog_manager.cc:5484] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 reported cstate change: term changed from 2 to 3, leader changed from 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) to ca177138f2c44d2eafa1646b795f7608 (127.0.99.136). New cstate: current_term: 3 leader_uuid: "ca177138f2c44d2eafa1646b795f7608" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:07.576597 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:07.585736 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:07.589587 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:07.591904 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:07.600111 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:07.602538 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:07.616936 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:07.630256 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:07.642081 11532 consensus_queue.cc:237] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:07.643568 13316 raft_consensus.cc:1261] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:00:07.644775 20731 consensus_queue.cc:1035] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:07.671634 20731 raft_consensus.cc:2932] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:07.675081 13316 raft_consensus.cc:2932] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:07.678498 11127 catalog_manager.cc:5484] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 6 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:07.684574 21012 ts_tablet_manager.cc:902] T 90f11b5693d7456d8a604e6c8726e69a P e5c03261d31c4d9a91077a17e1cd0336: Initiating tablet copy from peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 08:00:07.685137 21012 tablet_copy_client.cc:308] T 90f11b5693d7456d8a604e6c8726e69a P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37957
I20240501 08:00:07.685751 11551 tablet_copy_service.cc:140] P 22ec12774a4047a8b859743dc73f8ff6: Received BeginTabletCopySession request for tablet 90f11b5693d7456d8a604e6c8726e69a from peer e5c03261d31c4d9a91077a17e1cd0336 ({username='slave'} at 127.0.99.135:47673)
I20240501 08:00:07.685878 11551 tablet_copy_service.cc:161] P 22ec12774a4047a8b859743dc73f8ff6: Beginning new tablet copy session on tablet 90f11b5693d7456d8a604e6c8726e69a from peer e5c03261d31c4d9a91077a17e1cd0336 at {username='slave'} at 127.0.99.135:47673: session id = e5c03261d31c4d9a91077a17e1cd0336-90f11b5693d7456d8a604e6c8726e69a
I20240501 08:00:07.686990 11551 tablet_copy_source_session.cc:205] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:07.688452 21012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f11b5693d7456d8a604e6c8726e69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:07.690922 21012 tablet_copy_client.cc:783] T 90f11b5693d7456d8a604e6c8726e69a P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:07.691092 21012 tablet_copy_client.cc:647] T 90f11b5693d7456d8a604e6c8726e69a P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:07.693372 11532 consensus_queue.cc:237] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:07.693694 21012 tablet_copy_client.cc:523] T 90f11b5693d7456d8a604e6c8726e69a P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 08:00:07.695199 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd8348990f1b496bb62d8c0484193442. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:07.695716 13316 raft_consensus.cc:1261] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:00:07.696190 19840 consensus_queue.cc:1035] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:07.696936 21012 tablet_bootstrap.cc:492] T 90f11b5693d7456d8a604e6c8726e69a P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap starting.
I20240501 08:00:07.699316 19840 raft_consensus.cc:2932] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:07.700968 11248 tablet_service.cc:1514] Processing DeleteTablet for tablet bd8348990f1b496bb62d8c0484193442 with delete_type TABLET_DATA_TOMBSTONED (TS e2dafc1866314669ad0de8fb7bf7fd05 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42432
I20240501 08:00:07.701229 20627 tablet_replica.cc:330] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05: stopping tablet replica
I20240501 08:00:07.701375 20627 raft_consensus.cc:2229] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:07.701498 20627 raft_consensus.cc:2258] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:07.701478 13316 raft_consensus.cc:2932] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:07.701994 20627 ts_tablet_manager.cc:1875] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:07.703464 11114 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet bd8348990f1b496bb62d8c0484193442 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:00:07.704236 20627 ts_tablet_manager.cc:1888] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:00:07.704288 20627 log.cc:1196] T bd8348990f1b496bb62d8c0484193442 P e2dafc1866314669ad0de8fb7bf7fd05: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/bd8348990f1b496bb62d8c0484193442
I20240501 08:00:07.707060 11121 catalog_manager.cc:5484] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336 reported cstate change: config changed from index 7 to 8, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:07.713281 11113 catalog_manager.cc:4830] TS e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085): tablet bd8348990f1b496bb62d8c0484193442 (table rebalance_test_table_2 [id=69f109081c7449be8819974c5363fa32]) successfully deleted
I20240501 08:00:07.737660 21012 tablet_bootstrap.cc:492] T 90f11b5693d7456d8a604e6c8726e69a P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:07.738030 21012 tablet_bootstrap.cc:492] T 90f11b5693d7456d8a604e6c8726e69a P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap complete.
I20240501 08:00:07.738251 21012 ts_tablet_manager.cc:1367] T 90f11b5693d7456d8a604e6c8726e69a P e5c03261d31c4d9a91077a17e1cd0336: Time spent bootstrapping tablet: real 0.041s user 0.026s sys 0.000s
I20240501 08:00:07.738312 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:07.740563 21012 raft_consensus.cc:348] T 90f11b5693d7456d8a604e6c8726e69a P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:07.740707 21012 raft_consensus.cc:729] T 90f11b5693d7456d8a604e6c8726e69a P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c03261d31c4d9a91077a17e1cd0336, State: Initialized, Role: FOLLOWER
I20240501 08:00:07.741251 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:07.741966 21012 consensus_queue.cc:260] T 90f11b5693d7456d8a604e6c8726e69a P e5c03261d31c4d9a91077a17e1cd0336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:07.743001 21012 ts_tablet_manager.cc:1398] T 90f11b5693d7456d8a604e6c8726e69a P e5c03261d31c4d9a91077a17e1cd0336: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:00:07.747807 11551 tablet_copy_service.cc:342] P 22ec12774a4047a8b859743dc73f8ff6: Request end of tablet copy session e5c03261d31c4d9a91077a17e1cd0336-90f11b5693d7456d8a604e6c8726e69a received from {username='slave'} at 127.0.99.135:47673
I20240501 08:00:07.747933 11551 tablet_copy_service.cc:433] P 22ec12774a4047a8b859743dc73f8ff6: ending tablet copy session e5c03261d31c4d9a91077a17e1cd0336-90f11b5693d7456d8a604e6c8726e69a on tablet 90f11b5693d7456d8a604e6c8726e69a with peer e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:00:07.752185 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:07.756592 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:07.757385 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:07.773424 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:07.775930 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:07.782981 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:07.816422 13186 consensus_queue.cc:237] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:07.818540 11401 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer 53d1d2c0767b4f6195b69758a3dd6cf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:00:07.820056 20716 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:07.820734 13054 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Refusing update from remote peer 53d1d2c0767b4f6195b69758a3dd6cf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:00:07.821689 20716 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:07.823123 20716 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:07.826542 11401 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:07.830384 11120 catalog_manager.cc:5484] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 7 to 8, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:07.832691 13054 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:07.834733 11642 tablet_service.cc:1514] Processing DeleteTablet for tablet 25cee7e51c7c438aa721d71fb82dc71e with delete_type TABLET_DATA_TOMBSTONED (TS 603919bb02364c04b6c530245b8b02f2 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:53514
I20240501 08:00:07.843127 21097 tablet_replica.cc:330] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2: stopping tablet replica
I20240501 08:00:07.843308 21097 raft_consensus.cc:2229] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:07.843514 21097 raft_consensus.cc:2258] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:07.846278 21097 ts_tablet_manager.cc:1875] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:07.857645 21097 ts_tablet_manager.cc:1888] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240501 08:00:07.857740 21097 log.cc:1196] T 25cee7e51c7c438aa721d71fb82dc71e P 603919bb02364c04b6c530245b8b02f2: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/25cee7e51c7c438aa721d71fb82dc71e
I20240501 08:00:07.858328 11112 catalog_manager.cc:4830] TS 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): tablet 25cee7e51c7c438aa721d71fb82dc71e (table rebalance_test_table_2 [id=69f109081c7449be8819974c5363fa32]) successfully deleted
I20240501 08:00:07.871713 11268 consensus_queue.cc:237] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:07.873565 13446 raft_consensus.cc:1261] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:00:07.874125 20610 consensus_queue.cc:1035] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:07.874816 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:07.877573 20610 raft_consensus.cc:2932] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:07.880036 13446 raft_consensus.cc:2932] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:07.882794 11126 catalog_manager.cc:5484] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608 reported cstate change: config changed from index 6 to 7, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:07.885861 11268 consensus_queue.cc:237] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:07.886591 11510 tablet_service.cc:1514] Processing DeleteTablet for tablet b9431cf5853c4148ac38855b1c083d4f with delete_type TABLET_DATA_TOMBSTONED (TS 22ec12774a4047a8b859743dc73f8ff6 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52810
I20240501 08:00:07.886945 21159 tablet_replica.cc:330] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6: stopping tablet replica
I20240501 08:00:07.887084 21159 raft_consensus.cc:2229] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:07.887722 21159 raft_consensus.cc:2258] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240501 08:00:07.888375 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9431cf5853c4148ac38855b1c083d4f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:07.888559 13448 raft_consensus.cc:1261] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:00:07.889130 18953 consensus_queue.cc:1035] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:07.890239 21159 ts_tablet_manager.cc:1875] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:07.891266 18953 raft_consensus.cc:2932] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:07.893021 11113 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet b9431cf5853c4148ac38855b1c083d4f with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:00:07.893875 13448 raft_consensus.cc:2932] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:07.894647 11126 catalog_manager.cc:5484] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index 7 to 8, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New cstate: current_term: 2 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:07.896986 21159 ts_tablet_manager.cc:1888] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:00:07.897133 21159 log.cc:1196] T b9431cf5853c4148ac38855b1c083d4f P 22ec12774a4047a8b859743dc73f8ff6: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b9431cf5853c4148ac38855b1c083d4f
I20240501 08:00:07.898286 11114 catalog_manager.cc:4830] TS 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957): tablet b9431cf5853c4148ac38855b1c083d4f (table rebalance_test_table_1 [id=98fb02bdb5f54b1699749e3c8d588a5e]) successfully deleted
I20240501 08:00:07.907029 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:07.917702 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:07.917691 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:07.918543 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:07.919474 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:07.931226 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:07.937039 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:08.002912 11268 raft_consensus.cc:1261] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 3 FOLLOWER]: Refusing update from remote peer ca177138f2c44d2eafa1646b795f7608: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240501 08:00:08.003986 20947 consensus_queue.cc:1035] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:08.020758 11401 raft_consensus.cc:1261] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 3 FOLLOWER]: Refusing update from remote peer ca177138f2c44d2eafa1646b795f7608: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240501 08:00:08.022076 20947 consensus_queue.cc:1035] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:08.048198 13316 raft_consensus.cc:1203] T 90f11b5693d7456d8a604e6c8726e69a P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240501 08:00:08.060717 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:08.065508 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:08.072153 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:08.072292 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:08.104890 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:08.110601 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:08.114403 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:08.115563 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:08.144270 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:08.148074 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:08.157882 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:08.173439 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:08.178638 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:08.207698 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:08.209705 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:08.209741 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:08.211273 21361 ts_tablet_manager.cc:902] T bd8348990f1b496bb62d8c0484193442 P 60e8857abe7c41038af46ebb5a095a90: Initiating tablet copy from peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 08:00:08.211869 21361 tablet_copy_client.cc:308] T bd8348990f1b496bb62d8c0484193442 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37957
I20240501 08:00:08.212477 11551 tablet_copy_service.cc:140] P 22ec12774a4047a8b859743dc73f8ff6: Received BeginTabletCopySession request for tablet bd8348990f1b496bb62d8c0484193442 from peer 60e8857abe7c41038af46ebb5a095a90 ({username='slave'} at 127.0.99.133:59721)
I20240501 08:00:08.212594 11551 tablet_copy_service.cc:161] P 22ec12774a4047a8b859743dc73f8ff6: Beginning new tablet copy session on tablet bd8348990f1b496bb62d8c0484193442 from peer 60e8857abe7c41038af46ebb5a095a90 at {username='slave'} at 127.0.99.133:59721: session id = 60e8857abe7c41038af46ebb5a095a90-bd8348990f1b496bb62d8c0484193442
I20240501 08:00:08.213604 11551 tablet_copy_source_session.cc:205] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:08.214841 21361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8348990f1b496bb62d8c0484193442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:08.217548 21361 tablet_copy_client.cc:783] T bd8348990f1b496bb62d8c0484193442 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:08.217947 21361 tablet_copy_client.cc:647] T bd8348990f1b496bb62d8c0484193442 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:08.222208 21361 tablet_copy_client.cc:523] T bd8348990f1b496bb62d8c0484193442 P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:08.239598 21361 tablet_bootstrap.cc:492] T bd8348990f1b496bb62d8c0484193442 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap starting.
I20240501 08:00:08.261420 11531 consensus_queue.cc:237] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:08.262681 13054 raft_consensus.cc:1261] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:00:08.263118 20733 consensus_queue.cc:1035] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:08.271581 21421 raft_consensus.cc:2932] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:08.272188 13054 raft_consensus.cc:2932] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:08.275156 11127 catalog_manager.cc:5484] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90 reported cstate change: config changed from index 5 to 7, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:08.279240 11380 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e8550ea95984d08937b9b3e2af12bec with delete_type TABLET_DATA_TOMBSTONED (TS 655ebda1fb2c49c4af7087a760529835 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:43354
I20240501 08:00:08.280937 21361 tablet_bootstrap.cc:492] T bd8348990f1b496bb62d8c0484193442 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:08.280853 11531 consensus_queue.cc:237] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:08.281394 21361 tablet_bootstrap.cc:492] T bd8348990f1b496bb62d8c0484193442 P 60e8857abe7c41038af46ebb5a095a90: Bootstrap complete.
I20240501 08:00:08.281647 21361 ts_tablet_manager.cc:1367] T bd8348990f1b496bb62d8c0484193442 P 60e8857abe7c41038af46ebb5a095a90: Time spent bootstrapping tablet: real 0.042s user 0.024s sys 0.001s
I20240501 08:00:08.282199 21361 raft_consensus.cc:348] T bd8348990f1b496bb62d8c0484193442 P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:08.282392 21361 raft_consensus.cc:729] T bd8348990f1b496bb62d8c0484193442 P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e8857abe7c41038af46ebb5a095a90, State: Initialized, Role: FOLLOWER
W20240501 08:00:08.282399 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e8550ea95984d08937b9b3e2af12bec. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:08.282925 13054 raft_consensus.cc:1261] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:00:08.283390 20733 consensus_queue.cc:1035] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:08.285388 20733 raft_consensus.cc:2932] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:08.287950 13054 raft_consensus.cc:2932] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:08.290915 11127 catalog_manager.cc:5484] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90 reported cstate change: config changed from index 7 to 8, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:08.294087 21438 tablet_replica.cc:330] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: stopping tablet replica
I20240501 08:00:08.294308 21438 raft_consensus.cc:2229] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:08.294507 21438 raft_consensus.cc:2258] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:08.295101 21438 ts_tablet_manager.cc:1875] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:08.298540 21361 consensus_queue.cc:260] T bd8348990f1b496bb62d8c0484193442 P 60e8857abe7c41038af46ebb5a095a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:08.299050 21361 ts_tablet_manager.cc:1398] T bd8348990f1b496bb62d8c0484193442 P 60e8857abe7c41038af46ebb5a095a90: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20240501 08:00:08.299582 11551 tablet_copy_service.cc:342] P 22ec12774a4047a8b859743dc73f8ff6: Request end of tablet copy session 60e8857abe7c41038af46ebb5a095a90-bd8348990f1b496bb62d8c0484193442 received from {username='slave'} at 127.0.99.133:59721
I20240501 08:00:08.299664 11551 tablet_copy_service.cc:433] P 22ec12774a4047a8b859743dc73f8ff6: ending tablet copy session 60e8857abe7c41038af46ebb5a095a90-bd8348990f1b496bb62d8c0484193442 on tablet bd8348990f1b496bb62d8c0484193442 with peer 60e8857abe7c41038af46ebb5a095a90
I20240501 08:00:08.302559 11114 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 5e8550ea95984d08937b9b3e2af12bec with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:00:08.305985 21438 ts_tablet_manager.cc:1888] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:00:08.306198 21438 log.cc:1196] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5e8550ea95984d08937b9b3e2af12bec
I20240501 08:00:08.309000 11112 catalog_manager.cc:4830] TS 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575): tablet 5e8550ea95984d08937b9b3e2af12bec (table rebalance_test_table_7 [id=36e20074d8dc4402bd7d9b2039213af5]) successfully deleted
I20240501 08:00:08.357360 13186 consensus_queue.cc:237] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:08.359594 13054 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Refusing update from remote peer 53d1d2c0767b4f6195b69758a3dd6cf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:00:08.360504 21499 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:08.363493 21499 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:08.366865 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:08.367630 13054 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:08.371644 11128 catalog_manager.cc:5484] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90 reported cstate change: config changed from index 8 to 9, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:08.372241 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:08.378177 11380 tablet_service.cc:1514] Processing DeleteTablet for tablet 25cee7e51c7c438aa721d71fb82dc71e with delete_type TABLET_DATA_TOMBSTONED (TS 655ebda1fb2c49c4af7087a760529835 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43354
I20240501 08:00:08.388993 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:08.390965 21438 tablet_replica.cc:330] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835: stopping tablet replica
I20240501 08:00:08.391129 21438 raft_consensus.cc:2229] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:08.391335 21438 raft_consensus.cc:2258] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:08.392077 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:08.396323 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:08.397877 21438 ts_tablet_manager.cc:1875] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:08.405259 21438 ts_tablet_manager.cc:1888] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 08:00:08.405443 21438 log.cc:1196] T 25cee7e51c7c438aa721d71fb82dc71e P 655ebda1fb2c49c4af7087a760529835: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/25cee7e51c7c438aa721d71fb82dc71e
I20240501 08:00:08.406018 11112 catalog_manager.cc:4830] TS 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575): tablet 25cee7e51c7c438aa721d71fb82dc71e (table rebalance_test_table_2 [id=69f109081c7449be8819974c5363fa32]) successfully deleted
I20240501 08:00:08.409431 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:08.410552 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:08.410923 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:08.418511 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:08.421289 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:08.435057 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:08.441537 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:08.450584 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:08.450584 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:08.464756 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:08.466490 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:08.494953 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:08.503080 11642 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:00:08.508697 21361 ts_tablet_manager.cc:902] T b9431cf5853c4148ac38855b1c083d4f P 60e8857abe7c41038af46ebb5a095a90: Initiating tablet copy from peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 08:00:08.509214 21361 tablet_copy_client.cc:308] T b9431cf5853c4148ac38855b1c083d4f P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46085
I20240501 08:00:08.510600 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:08.510955 11290 tablet_copy_service.cc:140] P e2dafc1866314669ad0de8fb7bf7fd05: Received BeginTabletCopySession request for tablet b9431cf5853c4148ac38855b1c083d4f from peer 60e8857abe7c41038af46ebb5a095a90 ({username='slave'} at 127.0.99.133:60295)
I20240501 08:00:08.511049 11290 tablet_copy_service.cc:161] P e2dafc1866314669ad0de8fb7bf7fd05: Beginning new tablet copy session on tablet b9431cf5853c4148ac38855b1c083d4f from peer 60e8857abe7c41038af46ebb5a095a90 at {username='slave'} at 127.0.99.133:60295: session id = 60e8857abe7c41038af46ebb5a095a90-b9431cf5853c4148ac38855b1c083d4f
I20240501 08:00:08.511978 11290 tablet_copy_source_session.cc:205] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:08.523191 13186 consensus_queue.cc:237] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:08.525693 13054 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Refusing update from remote peer 53d1d2c0767b4f6195b69758a3dd6cf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:00:08.526187 21499 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:08.527556 21361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9431cf5853c4148ac38855b1c083d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:08.529470 21499 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:08.529897 13054 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:08.530311 21361 tablet_copy_client.cc:783] T b9431cf5853c4148ac38855b1c083d4f P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:08.530491 21361 tablet_copy_client.cc:647] T b9431cf5853c4148ac38855b1c083d4f P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:08.531584 11112 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 25cee7e51c7c438aa721d71fb82dc71e with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240501 08:00:08.531744 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25cee7e51c7c438aa721d71fb82dc71e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:08.532847 11124 catalog_manager.cc:5484] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 reported cstate change: config changed from index 9 to 10, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New cstate: current_term: 2 leader_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:08.534866 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:08.536957 21361 tablet_copy_client.cc:523] T b9431cf5853c4148ac38855b1c083d4f P 60e8857abe7c41038af46ebb5a095a90: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:08.537950 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:08.539837 21361 tablet_bootstrap.cc:492] T b9431cf5853c4148ac38855b1c083d4f P 60e8857abe7c41038af46ebb5a095a90: Bootstrap starting.
I20240501 08:00:08.540099 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:08.550030 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:08.557687 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:08.583401 13054 raft_consensus.cc:1203] T bd8348990f1b496bb62d8c0484193442 P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 08:00:08.595921 21361 tablet_bootstrap.cc:492] T b9431cf5853c4148ac38855b1c083d4f P 60e8857abe7c41038af46ebb5a095a90: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:08.596647 21361 tablet_bootstrap.cc:492] T b9431cf5853c4148ac38855b1c083d4f P 60e8857abe7c41038af46ebb5a095a90: Bootstrap complete.
I20240501 08:00:08.596858 21361 ts_tablet_manager.cc:1367] T b9431cf5853c4148ac38855b1c083d4f P 60e8857abe7c41038af46ebb5a095a90: Time spent bootstrapping tablet: real 0.057s user 0.026s sys 0.001s
I20240501 08:00:08.597388 21361 raft_consensus.cc:348] T b9431cf5853c4148ac38855b1c083d4f P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:08.597515 21361 raft_consensus.cc:729] T b9431cf5853c4148ac38855b1c083d4f P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e8857abe7c41038af46ebb5a095a90, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.597636 21361 consensus_queue.cc:260] T b9431cf5853c4148ac38855b1c083d4f P 60e8857abe7c41038af46ebb5a095a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:08.597930 21361 ts_tablet_manager.cc:1398] T b9431cf5853c4148ac38855b1c083d4f P 60e8857abe7c41038af46ebb5a095a90: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:08.598490 11290 tablet_copy_service.cc:342] P e2dafc1866314669ad0de8fb7bf7fd05: Request end of tablet copy session 60e8857abe7c41038af46ebb5a095a90-b9431cf5853c4148ac38855b1c083d4f received from {username='slave'} at 127.0.99.133:60295
I20240501 08:00:08.598569 11290 tablet_copy_service.cc:433] P e2dafc1866314669ad0de8fb7bf7fd05: ending tablet copy session 60e8857abe7c41038af46ebb5a095a90-b9431cf5853c4148ac38855b1c083d4f on tablet b9431cf5853c4148ac38855b1c083d4f with peer 60e8857abe7c41038af46ebb5a095a90
I20240501 08:00:08.660457 11268 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "867ba08a15b44d4c9e6fa92b0d518d1d"
dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44430
I20240501 08:00:08.660662 11268 raft_consensus.cc:595] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:00:08.696390 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:08.710594 11248 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:00:08.715880 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:08.719684 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:08.724507 18953 raft_consensus.cc:979] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05: : Instructing follower 603919bb02364c04b6c530245b8b02f2 to start an election
I20240501 08:00:08.724606 18953 raft_consensus.cc:1067] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Signalling peer 603919bb02364c04b6c530245b8b02f2 to start an election
I20240501 08:00:08.725651 11662 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "867ba08a15b44d4c9e6fa92b0d518d1d"
dest_uuid: "603919bb02364c04b6c530245b8b02f2"
from {username='slave'} at 127.0.99.129:41315
I20240501 08:00:08.725831 11662 raft_consensus.cc:482] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:00:08.725888 11662 raft_consensus.cc:3037] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:08.726977 11662 raft_consensus.cc:504] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:08.727489 11662 leader_election.cc:288] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 2 election: Requested vote from peers e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085), ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503)
I20240501 08:00:08.728267 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867ba08a15b44d4c9e6fa92b0d518d1d" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ca177138f2c44d2eafa1646b795f7608"
I20240501 08:00:08.728384 13446 raft_consensus.cc:3037] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:08.729415 13446 raft_consensus.cc:2454] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 2.
I20240501 08:00:08.730059 11599 leader_election.cc:302] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603919bb02364c04b6c530245b8b02f2, ca177138f2c44d2eafa1646b795f7608; no voters:
I20240501 08:00:08.730486 20952 raft_consensus.cc:2781] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:00:08.730671 20952 raft_consensus.cc:686] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 2 LEADER]: Becoming Leader. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Running, Role: LEADER
I20240501 08:00:08.730840 20952 consensus_queue.cc:237] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:08.731696 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:08.732772 11268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867ba08a15b44d4c9e6fa92b0d518d1d" candidate_uuid: "603919bb02364c04b6c530245b8b02f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e2dafc1866314669ad0de8fb7bf7fd05"
I20240501 08:00:08.732910 11268 raft_consensus.cc:3032] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:00:08.732950 11268 raft_consensus.cc:729] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Running, Role: LEADER
I20240501 08:00:08.733098 11268 consensus_queue.cc:260] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:08.733268 11268 raft_consensus.cc:3037] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:00:08.734385 11268 raft_consensus.cc:2454] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 603919bb02364c04b6c530245b8b02f2 in term 2.
I20240501 08:00:08.735709 11126 catalog_manager.cc:5484] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 reported cstate change: term changed from 1 to 2, leader changed from e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) to 603919bb02364c04b6c530245b8b02f2 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:08.764977 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:08.765911 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:08.774710 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:08.778126 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:08.784884 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:08.786607 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:08.794665 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:08.799520 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:08.809602 13446 consensus_queue.cc:237] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:08.810910 11401 raft_consensus.cc:1261] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer ca177138f2c44d2eafa1646b795f7608: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:00:08.811352 20329 consensus_queue.cc:1035] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:08.815730 20329 raft_consensus.cc:2932] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:08.818920 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:08.819247 11126 catalog_manager.cc:5484] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 reported cstate change: config changed from index 6 to 7, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "ca177138f2c44d2eafa1646b795f7608" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:08.820204 11400 raft_consensus.cc:2932] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:08.822587 11642 tablet_service.cc:1514] Processing DeleteTablet for tablet f2e63d24d7df4fa3b2b26695e70ebe9f with delete_type TABLET_DATA_TOMBSTONED (TS 603919bb02364c04b6c530245b8b02f2 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53514
I20240501 08:00:08.831317 13446 consensus_queue.cc:237] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:08.835702 11400 raft_consensus.cc:1261] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer ca177138f2c44d2eafa1646b795f7608: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:00:08.837309 20329 consensus_queue.cc:1035] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 08:00:08.839174 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2e63d24d7df4fa3b2b26695e70ebe9f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:08.839550 21679 tablet_replica.cc:330] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2: stopping tablet replica
I20240501 08:00:08.839824 21679 raft_consensus.cc:2229] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:08.840080 21679 raft_consensus.cc:2258] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:08.839998 20329 raft_consensus.cc:2932] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:08.840680 21679 ts_tablet_manager.cc:1875] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:08.842118 11115 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f2e63d24d7df4fa3b2b26695e70ebe9f with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:00:08.842646 11400 raft_consensus.cc:2932] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:08.842979 11126 catalog_manager.cc:5484] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 reported cstate change: config changed from index 7 to 8, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New cstate: current_term: 2 leader_uuid: "ca177138f2c44d2eafa1646b795f7608" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:08.846504 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:08.848735 21690 ts_tablet_manager.cc:902] T 5e8550ea95984d08937b9b3e2af12bec P ca177138f2c44d2eafa1646b795f7608: Initiating tablet copy from peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 08:00:08.849237 21690 tablet_copy_client.cc:308] T 5e8550ea95984d08937b9b3e2af12bec P ca177138f2c44d2eafa1646b795f7608: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37957
I20240501 08:00:08.850512 21679 ts_tablet_manager.cc:1888] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:00:08.850590 21679 log.cc:1196] T f2e63d24d7df4fa3b2b26695e70ebe9f P 603919bb02364c04b6c530245b8b02f2: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f2e63d24d7df4fa3b2b26695e70ebe9f
I20240501 08:00:08.852294 11112 catalog_manager.cc:4830] TS 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): tablet f2e63d24d7df4fa3b2b26695e70ebe9f (table rebalance_test_table_2 [id=69f109081c7449be8819974c5363fa32]) successfully deleted
I20240501 08:00:08.852911 11551 tablet_copy_service.cc:140] P 22ec12774a4047a8b859743dc73f8ff6: Received BeginTabletCopySession request for tablet 5e8550ea95984d08937b9b3e2af12bec from peer ca177138f2c44d2eafa1646b795f7608 ({username='slave'} at 127.0.99.136:39189)
I20240501 08:00:08.853022 11551 tablet_copy_service.cc:161] P 22ec12774a4047a8b859743dc73f8ff6: Beginning new tablet copy session on tablet 5e8550ea95984d08937b9b3e2af12bec from peer ca177138f2c44d2eafa1646b795f7608 at {username='slave'} at 127.0.99.136:39189: session id = ca177138f2c44d2eafa1646b795f7608-5e8550ea95984d08937b9b3e2af12bec
I20240501 08:00:08.854066 11551 tablet_copy_source_session.cc:205] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:08.864008 21690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8550ea95984d08937b9b3e2af12bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:08.866266 21690 tablet_copy_client.cc:783] T 5e8550ea95984d08937b9b3e2af12bec P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:08.870576 21690 tablet_copy_client.cc:647] T 5e8550ea95984d08937b9b3e2af12bec P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:08.875972 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:08.877041 21690 tablet_copy_client.cc:523] T 5e8550ea95984d08937b9b3e2af12bec P ca177138f2c44d2eafa1646b795f7608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:08.878796 21690 tablet_bootstrap.cc:492] T 5e8550ea95984d08937b9b3e2af12bec P ca177138f2c44d2eafa1646b795f7608: Bootstrap starting.
I20240501 08:00:08.891711 11268 consensus_queue.cc:237] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:08.895077 13186 raft_consensus.cc:1261] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:08.895131 11400 raft_consensus.cc:1261] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 08:00:08.895707 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea97f34590684bcf998228245dbcd9b0. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:08.895820 18953 consensus_queue.cc:1035] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:08.896251 18953 consensus_queue.cc:1035] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:08.903796 13446 consensus_queue.cc:237] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:08.905429 11268 raft_consensus.cc:1261] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 3 FOLLOWER]: Refusing update from remote peer ca177138f2c44d2eafa1646b795f7608: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240501 08:00:08.906620 21677 consensus_queue.cc:1035] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:08.909183 21677 raft_consensus.cc:2932] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:08.912104 11122 catalog_manager.cc:5484] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 reported cstate change: config changed from index 6 to 8, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New cstate: current_term: 3 leader_uuid: "ca177138f2c44d2eafa1646b795f7608" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:08.912695 11268 raft_consensus.cc:2932] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:08.916059 11380 tablet_service.cc:1514] Processing DeleteTablet for tablet 53193305f868433c852755e8a8a81766 with delete_type TABLET_DATA_TOMBSTONED (TS 655ebda1fb2c49c4af7087a760529835 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:43354
I20240501 08:00:08.915928 21735 raft_consensus.cc:2932] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:08.916255 13186 raft_consensus.cc:2932] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:08.917098 13446 consensus_queue.cc:237] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:08.917788 21438 tablet_replica.cc:330] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835: stopping tablet replica
I20240501 08:00:08.917939 21438 raft_consensus.cc:2229] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:08.918043 21438 raft_consensus.cc:2258] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:08.918823 21438 ts_tablet_manager.cc:1875] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:08.920068 11268 raft_consensus.cc:1261] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 3 FOLLOWER]: Refusing update from remote peer ca177138f2c44d2eafa1646b795f7608: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240501 08:00:08.920076 11400 raft_consensus.cc:2932] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:08.920912 11127 catalog_manager.cc:5484] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index 5 to 6, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:08.921030 21438 ts_tablet_manager.cc:1888] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.7
I20240501 08:00:08.921090 21438 log.cc:1196] T 53193305f868433c852755e8a8a81766 P 655ebda1fb2c49c4af7087a760529835: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/53193305f868433c852755e8a8a81766
I20240501 08:00:08.921757 21678 consensus_queue.cc:1035] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:08.921844 11112 catalog_manager.cc:4830] TS 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575): tablet 53193305f868433c852755e8a8a81766 (table rebalance_test_table_7 [id=36e20074d8dc4402bd7d9b2039213af5]) successfully deleted
I20240501 08:00:08.924242 21677 raft_consensus.cc:2932] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:08.924619 11268 raft_consensus.cc:2932] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:08.926663 11115 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 53193305f868433c852755e8a8a81766 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:00:08.928216 11126 catalog_manager.cc:5484] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index 8 to 9, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New cstate: current_term: 3 leader_uuid: "ca177138f2c44d2eafa1646b795f7608" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
W20240501 08:00:08.930207 13382 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771): Couldn't send request to peer 53d1d2c0767b4f6195b69758a3dd6cf9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53193305f868433c852755e8a8a81766. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:08.935966 21690 tablet_bootstrap.cc:492] T 5e8550ea95984d08937b9b3e2af12bec P ca177138f2c44d2eafa1646b795f7608: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:08.936286 21690 tablet_bootstrap.cc:492] T 5e8550ea95984d08937b9b3e2af12bec P ca177138f2c44d2eafa1646b795f7608: Bootstrap complete.
I20240501 08:00:08.936492 21690 ts_tablet_manager.cc:1367] T 5e8550ea95984d08937b9b3e2af12bec P ca177138f2c44d2eafa1646b795f7608: Time spent bootstrapping tablet: real 0.058s user 0.027s sys 0.001s
I20240501 08:00:08.937083 21690 raft_consensus.cc:348] T 5e8550ea95984d08937b9b3e2af12bec P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:08.937215 21690 raft_consensus.cc:729] T 5e8550ea95984d08937b9b3e2af12bec P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca177138f2c44d2eafa1646b795f7608, State: Initialized, Role: FOLLOWER
I20240501 08:00:08.937865 21690 consensus_queue.cc:260] T 5e8550ea95984d08937b9b3e2af12bec P ca177138f2c44d2eafa1646b795f7608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:08.952827 21690 ts_tablet_manager.cc:1398] T 5e8550ea95984d08937b9b3e2af12bec P ca177138f2c44d2eafa1646b795f7608: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240501 08:00:08.953470 11551 tablet_copy_service.cc:342] P 22ec12774a4047a8b859743dc73f8ff6: Request end of tablet copy session ca177138f2c44d2eafa1646b795f7608-5e8550ea95984d08937b9b3e2af12bec received from {username='slave'} at 127.0.99.136:39189
I20240501 08:00:08.953553 11551 tablet_copy_service.cc:433] P 22ec12774a4047a8b859743dc73f8ff6: ending tablet copy session ca177138f2c44d2eafa1646b795f7608-5e8550ea95984d08937b9b3e2af12bec on tablet 5e8550ea95984d08937b9b3e2af12bec with peer ca177138f2c44d2eafa1646b795f7608
I20240501 08:00:08.968649 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:08.971154 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:08.975936 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:08.976686 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:08.988209 11662 consensus_queue.cc:237] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 08:00:08.990074 13054 raft_consensus.cc:1261] T ed88ea311f474ab28cc6fbe6069258bf P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:08.990989 20952 consensus_queue.cc:1035] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:08.992058 11531 raft_consensus.cc:1261] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:08.993491 20952 consensus_queue.cc:1035] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:00:08.994871 11597 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085): Couldn't send request to peer e2dafc1866314669ad0de8fb7bf7fd05. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:08.995268 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:08.997454 13053 raft_consensus.cc:1203] T b9431cf5853c4148ac38855b1c083d4f P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 08:00:09.016264 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:09.023443 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:09.031829 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:09.050163 20952 raft_consensus.cc:2932] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:00:09.052729 11531 raft_consensus.cc:2932] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:00:09.055828 11127 catalog_manager.cc:5484] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 5 to 6, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:00:09.060876 13054 raft_consensus.cc:2932] T ed88ea311f474ab28cc6fbe6069258bf P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:00:09.132723 11268 consensus_queue.cc:237] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:09.136329 13054 raft_consensus.cc:1261] T c95c4f3ef08f44d1b69fb7b98426f5af P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:09.137444 11531 raft_consensus.cc:1261] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:00:09.138193 21652 consensus_queue.cc:1035] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:00:09.138945 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c95c4f3ef08f44d1b69fb7b98426f5af. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:09.141330 21762 consensus_queue.cc:1035] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:09.145444 21936 ts_tablet_manager.cc:902] T 25cee7e51c7c438aa721d71fb82dc71e P e5c03261d31c4d9a91077a17e1cd0336: Initiating tablet copy from peer 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771)
I20240501 08:00:09.145987 21936 tablet_copy_client.cc:308] T 25cee7e51c7c438aa721d71fb82dc71e P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:36771
I20240501 08:00:09.147787 21652 raft_consensus.cc:2932] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:09.151727 11531 raft_consensus.cc:2932] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:09.153439 13053 raft_consensus.cc:2932] T c95c4f3ef08f44d1b69fb7b98426f5af P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:09.156937 11125 catalog_manager.cc:5484] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 5 to 6, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:09.169422 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:09.177177 11268 raft_consensus.cc:1261] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:09.177866 21814 consensus_queue.cc:1035] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:09.178272 13206 tablet_copy_service.cc:140] P 53d1d2c0767b4f6195b69758a3dd6cf9: Received BeginTabletCopySession request for tablet 25cee7e51c7c438aa721d71fb82dc71e from peer e5c03261d31c4d9a91077a17e1cd0336 ({username='slave'} at 127.0.99.135:54791)
I20240501 08:00:09.180003 13206 tablet_copy_service.cc:161] P 53d1d2c0767b4f6195b69758a3dd6cf9: Beginning new tablet copy session on tablet 25cee7e51c7c438aa721d71fb82dc71e from peer e5c03261d31c4d9a91077a17e1cd0336 at {username='slave'} at 127.0.99.135:54791: session id = e5c03261d31c4d9a91077a17e1cd0336-25cee7e51c7c438aa721d71fb82dc71e
I20240501 08:00:09.181465 13206 tablet_copy_source_session.cc:205] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:09.182875 21936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25cee7e51c7c438aa721d71fb82dc71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.186704 21936 tablet_copy_client.cc:783] T 25cee7e51c7c438aa721d71fb82dc71e P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:09.186738 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:09.187002 21936 tablet_copy_client.cc:647] T 25cee7e51c7c438aa721d71fb82dc71e P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:09.189301 21936 tablet_copy_client.cc:523] T 25cee7e51c7c438aa721d71fb82dc71e P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:09.189483 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:09.190912 21936 tablet_bootstrap.cc:492] T 25cee7e51c7c438aa721d71fb82dc71e P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap starting.
I20240501 08:00:09.191264 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:09.197515 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:09.209360 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:09.209726 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:09.210582 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:09.215546 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:09.220772 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:09.223209 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:09.237031 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:09.238795 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:09.247931 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:09.266160 13446 raft_consensus.cc:1261] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:00:09.266685 21814 consensus_queue.cc:1035] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:00:09.272951 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:09.286666 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:09.348107 21936 tablet_bootstrap.cc:492] T 25cee7e51c7c438aa721d71fb82dc71e P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:09.357928 21936 tablet_bootstrap.cc:492] T 25cee7e51c7c438aa721d71fb82dc71e P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap complete.
I20240501 08:00:09.358314 21936 ts_tablet_manager.cc:1367] T 25cee7e51c7c438aa721d71fb82dc71e P e5c03261d31c4d9a91077a17e1cd0336: Time spent bootstrapping tablet: real 0.167s user 0.034s sys 0.000s
I20240501 08:00:09.359020 21936 raft_consensus.cc:348] T 25cee7e51c7c438aa721d71fb82dc71e P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:09.360311 21936 raft_consensus.cc:729] T 25cee7e51c7c438aa721d71fb82dc71e P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c03261d31c4d9a91077a17e1cd0336, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.360566 21936 consensus_queue.cc:260] T 25cee7e51c7c438aa721d71fb82dc71e P e5c03261d31c4d9a91077a17e1cd0336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:09.361131 21936 ts_tablet_manager.cc:1398] T 25cee7e51c7c438aa721d71fb82dc71e P e5c03261d31c4d9a91077a17e1cd0336: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:00:09.364009 13206 tablet_copy_service.cc:342] P 53d1d2c0767b4f6195b69758a3dd6cf9: Request end of tablet copy session e5c03261d31c4d9a91077a17e1cd0336-25cee7e51c7c438aa721d71fb82dc71e received from {username='slave'} at 127.0.99.135:54791
I20240501 08:00:09.364135 13206 tablet_copy_service.cc:433] P 53d1d2c0767b4f6195b69758a3dd6cf9: ending tablet copy session e5c03261d31c4d9a91077a17e1cd0336-25cee7e51c7c438aa721d71fb82dc71e on tablet 25cee7e51c7c438aa721d71fb82dc71e with peer e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:00:09.367328 21936 ts_tablet_manager.cc:902] T f2e63d24d7df4fa3b2b26695e70ebe9f P e5c03261d31c4d9a91077a17e1cd0336: Initiating tablet copy from peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503)
I20240501 08:00:09.368533 21936 tablet_copy_client.cc:308] T f2e63d24d7df4fa3b2b26695e70ebe9f P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:34503
I20240501 08:00:09.373459 13469 tablet_copy_service.cc:140] P ca177138f2c44d2eafa1646b795f7608: Received BeginTabletCopySession request for tablet f2e63d24d7df4fa3b2b26695e70ebe9f from peer e5c03261d31c4d9a91077a17e1cd0336 ({username='slave'} at 127.0.99.135:47875)
I20240501 08:00:09.373623 13469 tablet_copy_service.cc:161] P ca177138f2c44d2eafa1646b795f7608: Beginning new tablet copy session on tablet f2e63d24d7df4fa3b2b26695e70ebe9f from peer e5c03261d31c4d9a91077a17e1cd0336 at {username='slave'} at 127.0.99.135:47875: session id = e5c03261d31c4d9a91077a17e1cd0336-f2e63d24d7df4fa3b2b26695e70ebe9f
I20240501 08:00:09.375147 13469 tablet_copy_source_session.cc:205] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:09.376317 21936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2e63d24d7df4fa3b2b26695e70ebe9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.379112 21936 tablet_copy_client.cc:783] T f2e63d24d7df4fa3b2b26695e70ebe9f P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:09.379374 21936 tablet_copy_client.cc:647] T f2e63d24d7df4fa3b2b26695e70ebe9f P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:09.380976 21936 tablet_copy_client.cc:523] T f2e63d24d7df4fa3b2b26695e70ebe9f P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:09.382302 21936 tablet_bootstrap.cc:492] T f2e63d24d7df4fa3b2b26695e70ebe9f P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap starting.
I20240501 08:00:09.392724 11268 consensus_queue.cc:237] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:09.395752 13186 raft_consensus.cc:1261] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:00:09.396242 11531 consensus_queue.cc:237] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
W20240501 08:00:09.396495 11205 consensus_peers.cc:485] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17bb1ff829bc47a6a0bd6030517652aa. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:09.396800 11400 raft_consensus.cc:1261] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
W20240501 08:00:09.398313 11465 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): Couldn't send request to peer 603919bb02364c04b6c530245b8b02f2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:09.398736 13316 raft_consensus.cc:1261] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:00:09.398818 21735 consensus_queue.cc:1035] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:09.399183 21735 consensus_queue.cc:1035] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:09.399525 21987 consensus_queue.cc:1035] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:09.400204 13053 raft_consensus.cc:1261] T bd8348990f1b496bb62d8c0484193442 P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:00:09.400679 21987 consensus_queue.cc:1035] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:09.414583 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:09.418005 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:09.419327 21936 tablet_bootstrap.cc:492] T f2e63d24d7df4fa3b2b26695e70ebe9f P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:09.419631 21936 tablet_bootstrap.cc:492] T f2e63d24d7df4fa3b2b26695e70ebe9f P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap complete.
I20240501 08:00:09.419816 21936 ts_tablet_manager.cc:1367] T f2e63d24d7df4fa3b2b26695e70ebe9f P e5c03261d31c4d9a91077a17e1cd0336: Time spent bootstrapping tablet: real 0.038s user 0.034s sys 0.000s
I20240501 08:00:09.420325 21936 raft_consensus.cc:348] T f2e63d24d7df4fa3b2b26695e70ebe9f P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:09.420457 21936 raft_consensus.cc:729] T f2e63d24d7df4fa3b2b26695e70ebe9f P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c03261d31c4d9a91077a17e1cd0336, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.420591 21936 consensus_queue.cc:260] T f2e63d24d7df4fa3b2b26695e70ebe9f P e5c03261d31c4d9a91077a17e1cd0336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:09.421442 21936 ts_tablet_manager.cc:1398] T f2e63d24d7df4fa3b2b26695e70ebe9f P e5c03261d31c4d9a91077a17e1cd0336: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:09.422590 13469 tablet_copy_service.cc:342] P ca177138f2c44d2eafa1646b795f7608: Request end of tablet copy session e5c03261d31c4d9a91077a17e1cd0336-f2e63d24d7df4fa3b2b26695e70ebe9f received from {username='slave'} at 127.0.99.135:47875
I20240501 08:00:09.422683 13469 tablet_copy_service.cc:433] P ca177138f2c44d2eafa1646b795f7608: ending tablet copy session e5c03261d31c4d9a91077a17e1cd0336-f2e63d24d7df4fa3b2b26695e70ebe9f on tablet f2e63d24d7df4fa3b2b26695e70ebe9f with peer e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:00:09.424647 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:09.431010 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:09.432817 21987 raft_consensus.cc:2932] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:00:09.436079 11127 catalog_manager.cc:5484] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 8 to 9, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:00:09.439179 13053 raft_consensus.cc:2932] T bd8348990f1b496bb62d8c0484193442 P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:00:09.440564 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:09.442931 13316 raft_consensus.cc:2932] T bd8348990f1b496bb62d8c0484193442 P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:00:09.446863 13446 raft_consensus.cc:1203] T 5e8550ea95984d08937b9b3e2af12bec P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 08:00:09.460557 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:09.461753 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:09.486644 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:09.490491 21734 raft_consensus.cc:2932] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:09.490906 13186 raft_consensus.cc:2932] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:09.494946 11400 raft_consensus.cc:2932] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:09.499104 11120 catalog_manager.cc:5484] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New cstate: current_term: 2 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:09.506201 21936 ts_tablet_manager.cc:902] T ea97f34590684bcf998228245dbcd9b0 P e5c03261d31c4d9a91077a17e1cd0336: Initiating tablet copy from peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 08:00:09.506742 21936 tablet_copy_client.cc:308] T ea97f34590684bcf998228245dbcd9b0 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46085
I20240501 08:00:09.507254 11290 tablet_copy_service.cc:140] P e2dafc1866314669ad0de8fb7bf7fd05: Received BeginTabletCopySession request for tablet ea97f34590684bcf998228245dbcd9b0 from peer e5c03261d31c4d9a91077a17e1cd0336 ({username='slave'} at 127.0.99.135:33281)
I20240501 08:00:09.507364 11290 tablet_copy_service.cc:161] P e2dafc1866314669ad0de8fb7bf7fd05: Beginning new tablet copy session on tablet ea97f34590684bcf998228245dbcd9b0 from peer e5c03261d31c4d9a91077a17e1cd0336 at {username='slave'} at 127.0.99.135:33281: session id = e5c03261d31c4d9a91077a17e1cd0336-ea97f34590684bcf998228245dbcd9b0
I20240501 08:00:09.508301 11290 tablet_copy_source_session.cc:205] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:09.509783 21936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea97f34590684bcf998228245dbcd9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.512203 21936 tablet_copy_client.cc:783] T ea97f34590684bcf998228245dbcd9b0 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:09.512351 21936 tablet_copy_client.cc:647] T ea97f34590684bcf998228245dbcd9b0 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:09.513810 21936 tablet_copy_client.cc:523] T ea97f34590684bcf998228245dbcd9b0 P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:09.515333 21936 tablet_bootstrap.cc:492] T ea97f34590684bcf998228245dbcd9b0 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap starting.
I20240501 08:00:09.532658 22117 ts_tablet_manager.cc:902] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9: Initiating tablet copy from peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503)
I20240501 08:00:09.533183 22117 tablet_copy_client.cc:308] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:34503
I20240501 08:00:09.555131 13469 tablet_copy_service.cc:140] P ca177138f2c44d2eafa1646b795f7608: Received BeginTabletCopySession request for tablet 53193305f868433c852755e8a8a81766 from peer 53d1d2c0767b4f6195b69758a3dd6cf9 ({username='slave'} at 127.0.99.134:35373)
I20240501 08:00:09.555374 13469 tablet_copy_service.cc:161] P ca177138f2c44d2eafa1646b795f7608: Beginning new tablet copy session on tablet 53193305f868433c852755e8a8a81766 from peer 53d1d2c0767b4f6195b69758a3dd6cf9 at {username='slave'} at 127.0.99.134:35373: session id = 53d1d2c0767b4f6195b69758a3dd6cf9-53193305f868433c852755e8a8a81766
I20240501 08:00:09.557868 13469 tablet_copy_source_session.cc:205] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:09.559371 21936 tablet_bootstrap.cc:492] T ea97f34590684bcf998228245dbcd9b0 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:09.559768 22117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53193305f868433c852755e8a8a81766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.559788 21936 tablet_bootstrap.cc:492] T ea97f34590684bcf998228245dbcd9b0 P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap complete.
I20240501 08:00:09.559983 21936 ts_tablet_manager.cc:1367] T ea97f34590684bcf998228245dbcd9b0 P e5c03261d31c4d9a91077a17e1cd0336: Time spent bootstrapping tablet: real 0.045s user 0.029s sys 0.000s
I20240501 08:00:09.560632 21936 raft_consensus.cc:348] T ea97f34590684bcf998228245dbcd9b0 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:09.560835 21936 raft_consensus.cc:729] T ea97f34590684bcf998228245dbcd9b0 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c03261d31c4d9a91077a17e1cd0336, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.561038 21936 consensus_queue.cc:260] T ea97f34590684bcf998228245dbcd9b0 P e5c03261d31c4d9a91077a17e1cd0336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:09.561574 21936 ts_tablet_manager.cc:1398] T ea97f34590684bcf998228245dbcd9b0 P e5c03261d31c4d9a91077a17e1cd0336: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:00:09.565435 11290 tablet_copy_service.cc:342] P e2dafc1866314669ad0de8fb7bf7fd05: Request end of tablet copy session e5c03261d31c4d9a91077a17e1cd0336-ea97f34590684bcf998228245dbcd9b0 received from {username='slave'} at 127.0.99.135:33281
I20240501 08:00:09.565562 11290 tablet_copy_service.cc:433] P e2dafc1866314669ad0de8fb7bf7fd05: ending tablet copy session e5c03261d31c4d9a91077a17e1cd0336-ea97f34590684bcf998228245dbcd9b0 on tablet ea97f34590684bcf998228245dbcd9b0 with peer e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:00:09.567340 22117 tablet_copy_client.cc:783] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:09.567561 22117 tablet_copy_client.cc:647] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:09.567894 22166 ts_tablet_manager.cc:902] T c95c4f3ef08f44d1b69fb7b98426f5af P ca177138f2c44d2eafa1646b795f7608: Initiating tablet copy from peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 08:00:09.568421 22166 tablet_copy_client.cc:308] T c95c4f3ef08f44d1b69fb7b98426f5af P ca177138f2c44d2eafa1646b795f7608: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46085
I20240501 08:00:09.568909 11290 tablet_copy_service.cc:140] P e2dafc1866314669ad0de8fb7bf7fd05: Received BeginTabletCopySession request for tablet c95c4f3ef08f44d1b69fb7b98426f5af from peer ca177138f2c44d2eafa1646b795f7608 ({username='slave'} at 127.0.99.136:55185)
I20240501 08:00:09.569008 11290 tablet_copy_service.cc:161] P e2dafc1866314669ad0de8fb7bf7fd05: Beginning new tablet copy session on tablet c95c4f3ef08f44d1b69fb7b98426f5af from peer ca177138f2c44d2eafa1646b795f7608 at {username='slave'} at 127.0.99.136:55185: session id = ca177138f2c44d2eafa1646b795f7608-c95c4f3ef08f44d1b69fb7b98426f5af
I20240501 08:00:09.569919 11290 tablet_copy_source_session.cc:205] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:09.572278 22166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95c4f3ef08f44d1b69fb7b98426f5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.575110 22117 tablet_copy_client.cc:523] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:09.576531 22117 tablet_bootstrap.cc:492] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap starting.
I20240501 08:00:09.577235 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:09.582953 22178 ts_tablet_manager.cc:902] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: Initiating tablet copy from peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 08:00:09.583864 22178 tablet_copy_client.cc:308] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:44403
I20240501 08:00:09.584481 11683 tablet_copy_service.cc:140] P 603919bb02364c04b6c530245b8b02f2: Received BeginTabletCopySession request for tablet ed88ea311f474ab28cc6fbe6069258bf from peer e2dafc1866314669ad0de8fb7bf7fd05 ({username='slave'} at 127.0.99.129:41315)
I20240501 08:00:09.584597 11683 tablet_copy_service.cc:161] P 603919bb02364c04b6c530245b8b02f2: Beginning new tablet copy session on tablet ed88ea311f474ab28cc6fbe6069258bf from peer e2dafc1866314669ad0de8fb7bf7fd05 at {username='slave'} at 127.0.99.129:41315: session id = e2dafc1866314669ad0de8fb7bf7fd05-ed88ea311f474ab28cc6fbe6069258bf
I20240501 08:00:09.585472 11683 tablet_copy_source_session.cc:205] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:09.586719 22178 ts_tablet_manager.cc:1875] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:00:09.589440 22178 ts_tablet_manager.cc:1888] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 08:00:09.589633 13316 raft_consensus.cc:1203] T 25cee7e51c7c438aa721d71fb82dc71e P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240501 08:00:09.589644 22178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed88ea311f474ab28cc6fbe6069258bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.591281 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:09.593138 22178 tablet_copy_client.cc:783] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:09.593355 22178 tablet_copy_client.cc:647] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:09.594955 22178 tablet_copy_client.cc:523] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:09.597002 22178 tablet_bootstrap.cc:492] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 08:00:09.598934 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:09.600545 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:09.623234 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:09.626056 22166 tablet_copy_client.cc:783] T c95c4f3ef08f44d1b69fb7b98426f5af P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:09.627182 22166 tablet_copy_client.cc:647] T c95c4f3ef08f44d1b69fb7b98426f5af P ca177138f2c44d2eafa1646b795f7608: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:09.629182 22166 tablet_copy_client.cc:523] T c95c4f3ef08f44d1b69fb7b98426f5af P ca177138f2c44d2eafa1646b795f7608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:09.632931 22166 tablet_bootstrap.cc:492] T c95c4f3ef08f44d1b69fb7b98426f5af P ca177138f2c44d2eafa1646b795f7608: Bootstrap starting.
I20240501 08:00:09.636202 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:09.646276 22178 tablet_bootstrap.cc:492] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:09.646634 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:09.648162 22178 tablet_bootstrap.cc:492] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap complete.
I20240501 08:00:09.648416 22178 ts_tablet_manager.cc:1367] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.051s user 0.030s sys 0.005s
I20240501 08:00:09.657470 22178 raft_consensus.cc:348] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 08:00:09.657671 22178 raft_consensus.cc:729] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.663563 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:09.666615 22178 consensus_queue.cc:260] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 08:00:09.670056 22178 ts_tablet_manager.cc:1398] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240501 08:00:09.671196 11683 tablet_copy_service.cc:342] P 603919bb02364c04b6c530245b8b02f2: Request end of tablet copy session e2dafc1866314669ad0de8fb7bf7fd05-ed88ea311f474ab28cc6fbe6069258bf received from {username='slave'} at 127.0.99.129:41315
I20240501 08:00:09.671286 11683 tablet_copy_service.cc:433] P 603919bb02364c04b6c530245b8b02f2: ending tablet copy session e2dafc1866314669ad0de8fb7bf7fd05-ed88ea311f474ab28cc6fbe6069258bf on tablet ed88ea311f474ab28cc6fbe6069258bf with peer e2dafc1866314669ad0de8fb7bf7fd05
I20240501 08:00:09.713478 22117 tablet_bootstrap.cc:492] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:09.713922 22117 tablet_bootstrap.cc:492] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap complete.
I20240501 08:00:09.714469 22117 ts_tablet_manager.cc:1367] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent bootstrapping tablet: real 0.138s user 0.023s sys 0.007s
I20240501 08:00:09.714996 22117 raft_consensus.cc:348] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:09.715183 22117 raft_consensus.cc:729] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d1d2c0767b4f6195b69758a3dd6cf9, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.715361 22117 consensus_queue.cc:260] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:09.715703 22117 ts_tablet_manager.cc:1398] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:09.718377 13469 tablet_copy_service.cc:342] P ca177138f2c44d2eafa1646b795f7608: Request end of tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-53193305f868433c852755e8a8a81766 received from {username='slave'} at 127.0.99.134:35373
I20240501 08:00:09.718489 13469 tablet_copy_service.cc:433] P ca177138f2c44d2eafa1646b795f7608: ending tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-53193305f868433c852755e8a8a81766 on tablet 53193305f868433c852755e8a8a81766 with peer 53d1d2c0767b4f6195b69758a3dd6cf9
I20240501 08:00:09.729005 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:09.729848 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:09.731663 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:09.741926 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:09.747025 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:09.748245 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:09.769095 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:09.777112 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:09.813148 22166 tablet_bootstrap.cc:492] T c95c4f3ef08f44d1b69fb7b98426f5af P ca177138f2c44d2eafa1646b795f7608: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:09.813649 22166 tablet_bootstrap.cc:492] T c95c4f3ef08f44d1b69fb7b98426f5af P ca177138f2c44d2eafa1646b795f7608: Bootstrap complete.
I20240501 08:00:09.813903 22166 ts_tablet_manager.cc:1367] T c95c4f3ef08f44d1b69fb7b98426f5af P ca177138f2c44d2eafa1646b795f7608: Time spent bootstrapping tablet: real 0.181s user 0.025s sys 0.005s
I20240501 08:00:09.814482 22166 raft_consensus.cc:348] T c95c4f3ef08f44d1b69fb7b98426f5af P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:09.814620 22166 raft_consensus.cc:729] T c95c4f3ef08f44d1b69fb7b98426f5af P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca177138f2c44d2eafa1646b795f7608, State: Initialized, Role: FOLLOWER
I20240501 08:00:09.814786 22166 consensus_queue.cc:260] T c95c4f3ef08f44d1b69fb7b98426f5af P ca177138f2c44d2eafa1646b795f7608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:09.815171 22166 ts_tablet_manager.cc:1398] T c95c4f3ef08f44d1b69fb7b98426f5af P ca177138f2c44d2eafa1646b795f7608: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:09.815645 11290 tablet_copy_service.cc:342] P e2dafc1866314669ad0de8fb7bf7fd05: Request end of tablet copy session ca177138f2c44d2eafa1646b795f7608-c95c4f3ef08f44d1b69fb7b98426f5af received from {username='slave'} at 127.0.99.136:55185
I20240501 08:00:09.815734 11290 tablet_copy_service.cc:433] P e2dafc1866314669ad0de8fb7bf7fd05: ending tablet copy session ca177138f2c44d2eafa1646b795f7608-c95c4f3ef08f44d1b69fb7b98426f5af on tablet c95c4f3ef08f44d1b69fb7b98426f5af with peer ca177138f2c44d2eafa1646b795f7608
I20240501 08:00:09.822238 11661 consensus_queue.cc:237] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:09.824208 13186 raft_consensus.cc:1261] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:09.824698 21847 consensus_queue.cc:1035] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:09.824944 13446 raft_consensus.cc:1261] T f80db2e37bbc4716ad3a5200c61ee97b P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 08:00:09.825439 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f80db2e37bbc4716ad3a5200c61ee97b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:09.826524 21847 consensus_queue.cc:1035] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:09.832486 11661 consensus_queue.cc:237] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:09.832540 11400 consensus_queue.cc:237] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:09.833632 21847 raft_consensus.cc:2932] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:09.834388 13053 raft_consensus.cc:1261] T a1ebf29682024645978b3caa2d4988f3 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:09.834985 13446 raft_consensus.cc:1261] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:00:09.835275 13317 raft_consensus.cc:1261] T a1ebf29682024645978b3caa2d4988f3 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:09.835532 21864 consensus_queue.cc:1035] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:09.835873 13186 raft_consensus.cc:2932] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:09.836776 21847 consensus_queue.cc:1035] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:09.838150 21864 consensus_queue.cc:1035] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:09.838501 13448 raft_consensus.cc:2932] T f80db2e37bbc4716ad3a5200c61ee97b P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:09.840878 11123 catalog_manager.cc:5484] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 reported cstate change: config changed from index 7 to 8, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:09.844089 13446 raft_consensus.cc:2932] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:09.845628 21864 raft_consensus.cc:2932] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:09.846352 22274 raft_consensus.cc:2932] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:09.850026 13053 raft_consensus.cc:2932] T a1ebf29682024645978b3caa2d4988f3 P 60e8857abe7c41038af46ebb5a095a90 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:09.850291 11123 catalog_manager.cc:5484] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608 reported cstate change: config changed from index 5 to 7, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:09.855381 13317 raft_consensus.cc:2932] T a1ebf29682024645978b3caa2d4988f3 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:09.856052 11661 consensus_queue.cc:237] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:09.858556 11127 catalog_manager.cc:5484] T a1ebf29682024645978b3caa2d4988f3 P e5c03261d31c4d9a91077a17e1cd0336 reported cstate change: config changed from index 6 to 7, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:09.859637 11248 tablet_service.cc:1514] Processing DeleteTablet for tablet 867ba08a15b44d4c9e6fa92b0d518d1d with delete_type TABLET_DATA_TOMBSTONED (TS e2dafc1866314669ad0de8fb7bf7fd05 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42432
W20240501 08:00:09.859901 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 867ba08a15b44d4c9e6fa92b0d518d1d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:09.860572 13446 raft_consensus.cc:1261] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:00:09.861006 22269 consensus_queue.cc:1035] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:09.863039 22269 raft_consensus.cc:2932] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:09.864301 22289 tablet_replica.cc:330] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05: stopping tablet replica
I20240501 08:00:09.864804 22289 raft_consensus.cc:2229] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:09.865058 22289 raft_consensus.cc:2258] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:09.867336 13446 raft_consensus.cc:2932] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:09.869921 11642 tablet_service.cc:1514] Processing DeleteTablet for tablet a1ebf29682024645978b3caa2d4988f3 with delete_type TABLET_DATA_TOMBSTONED (TS 603919bb02364c04b6c530245b8b02f2 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53514
I20240501 08:00:09.870012 11112 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 867ba08a15b44d4c9e6fa92b0d518d1d with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:00:09.870162 11128 catalog_manager.cc:5484] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608 reported cstate change: config changed from index 7 to 8, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) added. New cstate: current_term: 2 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:00:09.870863 22309 tablet_replica.cc:330] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2: stopping tablet replica
I20240501 08:00:09.871065 22309 raft_consensus.cc:2229] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:09.871266 22289 ts_tablet_manager.cc:1875] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:09.871603 22309 raft_consensus.cc:2258] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:09.874258 22289 ts_tablet_manager.cc:1888] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:00:09.874368 22289 log.cc:1196] T 867ba08a15b44d4c9e6fa92b0d518d1d P e2dafc1866314669ad0de8fb7bf7fd05: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/867ba08a15b44d4c9e6fa92b0d518d1d
I20240501 08:00:09.874936 11113 catalog_manager.cc:4830] TS e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085): tablet 867ba08a15b44d4c9e6fa92b0d518d1d (table rebalance_test_table_3 [id=6447707eb34f45d7a8d3704cd09d7786]) successfully deleted
I20240501 08:00:09.872354 22309 ts_tablet_manager.cc:1875] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:09.879009 22309 ts_tablet_manager.cc:1888] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:00:09.879218 22309 log.cc:1196] T a1ebf29682024645978b3caa2d4988f3 P 603919bb02364c04b6c530245b8b02f2: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a1ebf29682024645978b3caa2d4988f3
I20240501 08:00:09.879932 11112 catalog_manager.cc:4830] TS 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): tablet a1ebf29682024645978b3caa2d4988f3 (table rebalance_test_table_0 [id=24b4f6e8cc8c44f6961591cac7beacb4]) successfully deleted
I20240501 08:00:09.891094 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:09.894838 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:09.899190 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:09.902864 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:09.910657 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:09.915580 13317 raft_consensus.cc:1203] T f2e63d24d7df4fa3b2b26695e70ebe9f P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 08:00:09.922793 13186 raft_consensus.cc:1203] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.8->[3.9-3.9] Dedup: 3.9->[]
I20240501 08:00:09.926875 22349 ts_tablet_manager.cc:902] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: Initiating tablet copy from peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 08:00:09.927716 22349 tablet_copy_client.cc:308] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37957
I20240501 08:00:09.928395 11551 tablet_copy_service.cc:140] P 22ec12774a4047a8b859743dc73f8ff6: Received BeginTabletCopySession request for tablet bd8348990f1b496bb62d8c0484193442 from peer 603919bb02364c04b6c530245b8b02f2 ({username='slave'} at 127.0.99.132:54785)
I20240501 08:00:09.928530 11551 tablet_copy_service.cc:161] P 22ec12774a4047a8b859743dc73f8ff6: Beginning new tablet copy session on tablet bd8348990f1b496bb62d8c0484193442 from peer 603919bb02364c04b6c530245b8b02f2 at {username='slave'} at 127.0.99.132:54785: session id = 603919bb02364c04b6c530245b8b02f2-bd8348990f1b496bb62d8c0484193442
I20240501 08:00:09.929577 11551 tablet_copy_source_session.cc:205] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:09.930783 22349 ts_tablet_manager.cc:1875] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:00:09.933367 22349 ts_tablet_manager.cc:1888] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.5
I20240501 08:00:09.933727 22349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8348990f1b496bb62d8c0484193442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:09.935797 22349 tablet_copy_client.cc:783] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:09.936039 22349 tablet_copy_client.cc:647] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:09.939656 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:09.946628 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:09.950043 13186 consensus_queue.cc:237] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 08:00:09.951906 13053 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Refusing update from remote peer 53d1d2c0767b4f6195b69758a3dd6cf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 08:00:09.952458 21529 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:09.954977 13317 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Refusing update from remote peer 53d1d2c0767b4f6195b69758a3dd6cf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 08:00:09.955338 22366 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:09.958241 22349 tablet_copy_client.cc:523] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:09.959939 22349 tablet_bootstrap.cc:492] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 08:00:09.962888 13316 raft_consensus.cc:1203] T ea97f34590684bcf998228245dbcd9b0 P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
W20240501 08:00:09.963351 13120 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085): Couldn't send request to peer e2dafc1866314669ad0de8fb7bf7fd05. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:09.983426 11661 consensus_queue.cc:237] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 08:00:09.985042 13186 raft_consensus.cc:1261] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:09.985111 13446 raft_consensus.cc:1261] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:00:09.985532 22275 consensus_queue.cc:1035] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 08:00:09.985759 11598 consensus_peers.cc:485] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 -> Peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957): Couldn't send request to peer 22ec12774a4047a8b859743dc73f8ff6. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:09.985760 22274 consensus_queue.cc:1035] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:09.987814 22366 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:00:09.990077 13053 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:00:09.991441 11120 catalog_manager.cc:5484] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 reported cstate change: config changed from index 10 to 11, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) added. New cstate: current_term: 2 leader_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:00:09.993834 13031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:09.995571 13317 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P e5c03261d31c4d9a91077a17e1cd0336 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:00:09.998200 11268 raft_consensus.cc:1203] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 08:00:09.999692 22349 tablet_bootstrap.cc:492] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:09.999997 22349 tablet_bootstrap.cc:492] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: Bootstrap complete.
I20240501 08:00:10.000183 22349 ts_tablet_manager.cc:1367] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.040s user 0.024s sys 0.002s
I20240501 08:00:10.000483 13446 raft_consensus.cc:2932] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 08:00:10.000638 22349 raft_consensus.cc:348] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:00:10.000763 22349 raft_consensus.cc:729] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.000882 22349 consensus_queue.cc:260] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:00:10.001169 22349 ts_tablet_manager.cc:1398] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:00:10.002676 11551 tablet_copy_service.cc:342] P 22ec12774a4047a8b859743dc73f8ff6: Request end of tablet copy session 603919bb02364c04b6c530245b8b02f2-bd8348990f1b496bb62d8c0484193442 received from {username='slave'} at 127.0.99.132:54785
I20240501 08:00:10.002761 11551 tablet_copy_service.cc:433] P 22ec12774a4047a8b859743dc73f8ff6: ending tablet copy session 603919bb02364c04b6c530245b8b02f2-bd8348990f1b496bb62d8c0484193442 on tablet bd8348990f1b496bb62d8c0484193442 with peer 603919bb02364c04b6c530245b8b02f2
I20240501 08:00:10.002871 22274 raft_consensus.cc:2932] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 08:00:10.007162 11127 catalog_manager.cc:5484] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 reported cstate change: config changed from index 7 to 8, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 08:00:10.007329 13186 raft_consensus.cc:2932] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 08:00:10.008296 22393 ts_tablet_manager.cc:902] T 17bb1ff829bc47a6a0bd6030517652aa P 60e8857abe7c41038af46ebb5a095a90: Initiating tablet copy from peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
F20240501 08:00:10.008524 22393 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-164fd265561e47128e7ff585b8cfab07.queue_time_us, tablet-download-pool-164fd265561e47128e7ff585b8cfab07.queue_time_us) (tablet-download-pool-164fd265561e47128e7ff585b8cfab07.run_cpu_time_us, tablet-download-pool-164fd265561e47128e7ff585b8cfab07.run_cpu_time_us) (tablet-download-pool-164fd265561e47128e7ff585b8cfab07.run_wall_time_us, tablet-download-pool-164fd265561e47128e7ff585b8cfab07.run_wall_time_us) (tablet-download-pool-17bb1ff829bc47a6a0bd6030517652aa.queue_time_us, tablet-download-pool-17bb1ff829bc47a6a0bd6030517652aa.queue_time_us) (tablet-download-pool-25cee7e51c7c438aa721d71fb82dc71e.queue_time_us, tablet-download-pool-25cee7e51c7c438aa721d71fb82dc71e.queue_time_us) (tablet-download-pool-25cee7e51c7c438aa721d71fb82dc71e.run_cpu_time_us, tablet-download-pool-25cee7e51c7c438aa721d71fb82dc71e.run_cpu_time_us) (tablet-download-pool-25cee7e51c7c438aa721d71fb82dc71e.run_wall_time_us, tablet-download-pool-25cee7e51c7c438aa721d71fb82dc71e.run_wall_time_us) (tablet-download-pool-3f6e4ae97e884219ac7d0d94997d9cc9.queue_time_us, tablet-download-pool-3f6e4ae97e884219ac7d0d94997d9cc9.queue_time_us) (tablet-download-pool-3f6e4ae97e884219ac7d0d94997d9cc9.run_cpu_time_us, tablet-download-pool-3f6e4ae97e884219ac7d0d94997d9cc9.run_cpu_time_us) (tablet-download-pool-3f6e4ae97e884219ac7d0d94997d9cc9.run_wall_time_us, tablet-download-pool-3f6e4ae97e884219ac7d0d94997d9cc9.run_wall_time_us) (tablet-download-pool-5e8550ea95984d08937b9b3e2af12bec.queue_time_us, tablet-download-pool-5e8550ea95984d08937b9b3e2af12bec.queue_time_us) (tablet-download-pool-5e8550ea95984d08937b9b3e2af12bec.run_cpu_time_us, tablet-download-pool-5e8550ea95984d08937b9b3e2af12bec.run_cpu_time_us) (tablet-download-pool-5e8550ea95984d08937b9b3e2af12bec.run_wall_time_us, tablet-download-pool-5e8550ea95984d08937b9b3e2af12bec.run_wall_time_us) (tablet-download-pool-64613ea537c842f29d810e829278394e.queue_time_us, tablet-download-pool-64613ea537c842f29d810e829278394e.queue_time_us) (tablet-download-pool-64613ea537c842f29d810e829278394e.run_cpu_time_us, tablet-download-pool-64613ea537c842f29d810e829278394e.run_cpu_time_us) (tablet-download-pool-64613ea537c842f29d810e829278394e.run_wall_time_us, tablet-download-pool-64613ea537c842f29d810e829278394e.run_wall_time_us) (tablet-download-pool-6d214be86534458091f54a386d7d3334.queue_time_us, tablet-download-pool-6d214be86534458091f54a386d7d3334.queue_time_us) (tablet-download-pool-6d214be86534458091f54a386d7d3334.run_cpu_time_us, tablet-download-pool-6d214be86534458091f54a386d7d3334.run_cpu_time_us) (tablet-download-pool-6d214be86534458091f54a386d7d3334.run_wall_time_us, tablet-download-pool-6d214be86534458091f54a386d7d3334.run_wall_time_us) (tablet-download-pool-7a266f266c3d4fd49fa8b325f1045d61.queue_time_us, tablet-download-pool-7a266f266c3d4fd49fa8b325f1045d61.queue_time_us) (tablet-download-pool-7a266f266c3d4fd49fa8b325f1045d61.run_cpu_time_us, tablet-download-pool-7a266f266c3d4fd49fa8b325f1045d61.run_cpu_time_us) (tablet-download-pool-7a266f266c3d4fd49fa8b325f1045d61.run_wall_time_us, tablet-download-pool-7a266f266c3d4fd49fa8b325f1045d61.run_wall_time_us) (tablet-download-pool-a1ebf29682024645978b3caa2d4988f3.queue_time_us, tablet-download-pool-a1ebf29682024645978b3caa2d4988f3.queue_time_us) (tablet-download-pool-a1ebf29682024645978b3caa2d4988f3.run_cpu_time_us, tablet-download-pool-a1ebf29682024645978b3caa2d4988f3.run_cpu_time_us) (tablet-download-pool-a1ebf29682024645978b3caa2d4988f3.run_wall_time_us, tablet-download-pool-a1ebf29682024645978b3caa2d4988f3.run_wall_time_us) (tablet-download-pool-b21b6e2023e842739a7c0cc946e7c2fd.queue_time_us, tablet-download-pool-b21b6e2023e842739a7c0cc946e7c2fd.queue_time_us) (tablet-download-pool-b21b6e2023e842739a7c0cc946e7c2fd.run_cpu_time_us, tablet-download-pool-b21b6e2023e842739a7c0cc946e7c2fd.run_cpu_time_us) (tablet-download-pool-b21b6e2023e842739a7c0cc946e7c2fd.run_wall_time_us, tablet-download-pool-b21b6e2023e842739a7c0cc946e7c2fd.run_wall_time_us) (tablet-download-pool-b4707b952d174a9bb877ad8bd7a11268.queue_time_us, tablet-download-pool-b4707b952d174a9bb877ad8bd7a11268.queue_time_us) (tablet-download-pool-b4707b952d174a9bb877ad8bd7a11268.run_cpu_time_us, tablet-download-pool-b4707b952d174a9bb877ad8bd7a11268.run_cpu_time_us) (tablet-download-pool-b4707b952d174a9bb877ad8bd7a11268.run_wall_time_us, tablet-download-pool-b4707b952d174a9bb877ad8bd7a11268.run_wall_time_us) (tablet-download-pool-b9431cf5853c4148ac38855b1c083d4f.queue_time_us, tablet-download-pool-b9431cf5853c4148ac38855b1c083d4f.queue_time_us) (tablet-download-pool-b9431cf5853c4148ac38855b1c083d4f.run_cpu_time_us, tablet-download-pool-b9431cf5853c4148ac38855b1c083d4f.run_cpu_time_us) (tablet-download-pool-b9431cf5853c4148ac38855b1c083d4f.run_wall_time_us, tablet-download-pool-b9431cf5853c4148ac38855b1c083d4f.run_wall_time_us) (tablet-download-pool-bd8348990f1b496bb62d8c0484193442.queue_time_us, tablet-download-pool-bd8348990f1b496bb62d8c0484193442.queue_time_us) (tablet-download-pool-bd8348990f1b496bb62d8c0484193442.run_cpu_time_us, tablet-download-pool-bd8348990f1b496bb62d8c0484193442.run_cpu_time_us) (tablet-download-pool-bd8348990f1b496bb62d8c0484193442.run_wall_time_us, tablet-download-pool-bd8348990f1b496bb62d8c0484193442.run_wall_time_us) (tablet-download-pool-c95c4f3ef08f44d1b69fb7b98426f5af.queue_time_us, tablet-download-pool-c95c4f3ef08f44d1b69fb7b98426f5af.queue_time_us) (tablet-download-pool-c95c4f3ef08f44d1b69fb7b98426f5af.run_cpu_time_us, tablet-download-pool-c95c4f3ef08f44d1b69fb7b98426f5af.run_cpu_time_us) (tablet-download-pool-c95c4f3ef08f44d1b69fb7b98426f5af.run_wall_time_us, tablet-download-pool-c95c4f3ef08f44d1b69fb7b98426f5af.run_wall_time_us) (tablet-download-pool-ed88ea311f474ab28cc6fbe6069258bf.queue_time_us, tablet-download-pool-ed88ea311f474ab28cc6fbe6069258bf.queue_time_us) (tablet-download-pool-ed88ea311f474ab28cc6fbe6069258bf.run_cpu_time_us, tablet-download-pool-ed88ea311f474ab28cc6fbe6069258bf.run_cpu_time_us) (tablet-download-pool-ed88ea311f474ab28cc6fbe6069258bf.run_wall_time_us, tablet-download-pool-ed88ea311f474ab28cc6fbe6069258bf.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f659d7d6dcd google::LogMessage::Fail() at ??:0
@ 0x7f659d7dab93 google::LogMessage::SendToLog() at ??:0
@ 0x7f659d7d67cc google::LogMessage::Flush() at ??:0
@ 0x7f659d7d7f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f659ec40044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f659ec24f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f659ec22efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f65a17d577a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f65a17dea45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f65a185f76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f65a18734f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f65a185e226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f65a186dc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x558be1c4a75c std::function<>::operator()() at ??:0
@ 0x7f659ec27e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f659ec286ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f659ec2a03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x558be1c4a75c std::function<>::operator()() at ??:0
@ 0x7f659ec17807 kudu::Thread::SuperviseThread() at ??:0
I20240501 08:00:10.084867 11531 consensus_queue.cc:237] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:00:10.086504 13052 raft_consensus.cc:1261] T 5e8550ea95984d08937b9b3e2af12bec P 60e8857abe7c41038af46ebb5a095a90 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:00:10.086637 13446 raft_consensus.cc:1261] T 5e8550ea95984d08937b9b3e2af12bec P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:00:10.086956 21989 consensus_queue.cc:1035] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:10.087241 21986 consensus_queue.cc:1035] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 08:00:10.087778 11465 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): Couldn't send request to peer 603919bb02364c04b6c530245b8b02f2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:10.091063 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
@ 0x7f659f4556db start_thread at ??:0
@ 0x7f659c98d71f clone at ??:0
I20240501 08:00:10.115051 21986 raft_consensus.cc:2932] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:00:10.118842 11127 catalog_manager.cc:5484] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 8 to 9, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:00:10.119922 13446 raft_consensus.cc:2932] T 5e8550ea95984d08937b9b3e2af12bec P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:00:10.123306 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:10.143983 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:10.153650 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:10.163312 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:10.175115 13446 raft_consensus.cc:1203] T c95c4f3ef08f44d1b69fb7b98426f5af P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 08:00:10.180720 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:10.190817 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:10.192340 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:10.200256 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:10.205751 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:10.211920 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:10.219679 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:10.231500 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:10.270186 11400 consensus_queue.cc:237] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 08:00:10.272498 13446 raft_consensus.cc:1261] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 08:00:10.273075 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957): Couldn't send request to peer 22ec12774a4047a8b859743dc73f8ff6. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:10.273358 21864 consensus_queue.cc:1035] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:00:10.281733 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:10.282688 22540 ts_tablet_manager.cc:902] T f80db2e37bbc4716ad3a5200c61ee97b P e5c03261d31c4d9a91077a17e1cd0336: Initiating tablet copy from peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 08:00:10.291546 22540 tablet_copy_client.cc:308] T f80db2e37bbc4716ad3a5200c61ee97b P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:44403
I20240501 08:00:10.292198 11683 tablet_copy_service.cc:140] P 603919bb02364c04b6c530245b8b02f2: Received BeginTabletCopySession request for tablet f80db2e37bbc4716ad3a5200c61ee97b from peer e5c03261d31c4d9a91077a17e1cd0336 ({username='slave'} at 127.0.99.135:57735)
I20240501 08:00:10.292335 11683 tablet_copy_service.cc:161] P 603919bb02364c04b6c530245b8b02f2: Beginning new tablet copy session on tablet f80db2e37bbc4716ad3a5200c61ee97b from peer e5c03261d31c4d9a91077a17e1cd0336 at {username='slave'} at 127.0.99.135:57735: session id = e5c03261d31c4d9a91077a17e1cd0336-f80db2e37bbc4716ad3a5200c61ee97b
I20240501 08:00:10.292639 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:10.293272 11683 tablet_copy_source_session.cc:205] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:10.298408 22540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80db2e37bbc4716ad3a5200c61ee97b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.318336 22540 tablet_copy_client.cc:783] T f80db2e37bbc4716ad3a5200c61ee97b P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:10.318611 22540 tablet_copy_client.cc:647] T f80db2e37bbc4716ad3a5200c61ee97b P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:10.320717 22540 tablet_copy_client.cc:523] T f80db2e37bbc4716ad3a5200c61ee97b P e5c03261d31c4d9a91077a17e1cd0336: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:10.322365 22540 tablet_bootstrap.cc:492] T f80db2e37bbc4716ad3a5200c61ee97b P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap starting.
I20240501 08:00:10.326311 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:10.336740 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:10.344853 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:10.345906 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:10.387724 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:10.389410 13296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:00:10.410503 22540 tablet_bootstrap.cc:492] T f80db2e37bbc4716ad3a5200c61ee97b P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:10.410869 22540 tablet_bootstrap.cc:492] T f80db2e37bbc4716ad3a5200c61ee97b P e5c03261d31c4d9a91077a17e1cd0336: Bootstrap complete.
I20240501 08:00:10.411068 22540 ts_tablet_manager.cc:1367] T f80db2e37bbc4716ad3a5200c61ee97b P e5c03261d31c4d9a91077a17e1cd0336: Time spent bootstrapping tablet: real 0.089s user 0.030s sys 0.000s
I20240501 08:00:10.411764 22540 raft_consensus.cc:348] T f80db2e37bbc4716ad3a5200c61ee97b P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:10.411883 22540 raft_consensus.cc:729] T f80db2e37bbc4716ad3a5200c61ee97b P e5c03261d31c4d9a91077a17e1cd0336 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c03261d31c4d9a91077a17e1cd0336, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.412014 22540 consensus_queue.cc:260] T f80db2e37bbc4716ad3a5200c61ee97b P e5c03261d31c4d9a91077a17e1cd0336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:00:10.412011 11661 raft_consensus.cc:1203] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240501 08:00:10.412364 22540 ts_tablet_manager.cc:1398] T f80db2e37bbc4716ad3a5200c61ee97b P e5c03261d31c4d9a91077a17e1cd0336: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 08:00:10.413506 11683 tablet_copy_service.cc:342] P 603919bb02364c04b6c530245b8b02f2: Request end of tablet copy session e5c03261d31c4d9a91077a17e1cd0336-f80db2e37bbc4716ad3a5200c61ee97b received from {username='slave'} at 127.0.99.135:57735
I20240501 08:00:10.413574 11683 tablet_copy_service.cc:433] P 603919bb02364c04b6c530245b8b02f2: ending tablet copy session e5c03261d31c4d9a91077a17e1cd0336-f80db2e37bbc4716ad3a5200c61ee97b on tablet f80db2e37bbc4716ad3a5200c61ee97b with peer e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:00:10.417992 22540 ts_tablet_manager.cc:902] T 867ba08a15b44d4c9e6fa92b0d518d1d P e5c03261d31c4d9a91077a17e1cd0336: Initiating tablet copy from peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
F20240501 08:00:10.418165 22540 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-18b146cc4e164e1d9f9f827217c9b3ab.queue_time_us, tablet-download-pool-18b146cc4e164e1d9f9f827217c9b3ab.queue_time_us) (tablet-download-pool-18b146cc4e164e1d9f9f827217c9b3ab.run_cpu_time_us, tablet-download-pool-18b146cc4e164e1d9f9f827217c9b3ab.run_cpu_time_us) (tablet-download-pool-18b146cc4e164e1d9f9f827217c9b3ab.run_wall_time_us, tablet-download-pool-18b146cc4e164e1d9f9f827217c9b3ab.run_wall_time_us) (tablet-download-pool-25cee7e51c7c438aa721d71fb82dc71e.queue_time_us, tablet-download-pool-25cee7e51c7c438aa721d71fb82dc71e.queue_time_us) (tablet-download-pool-25cee7e51c7c438aa721d71fb82dc71e.run_cpu_time_us, tablet-download-pool-25cee7e51c7c438aa721d71fb82dc71e.run_cpu_time_us) (tablet-download-pool-25cee7e51c7c438aa721d71fb82dc71e.run_wall_time_us, tablet-download-pool-25cee7e51c7c438aa721d71fb82dc71e.run_wall_time_us) (tablet-download-pool-48b0ff0769dc426190f038811155c741.queue_time_us, tablet-download-pool-48b0ff0769dc426190f038811155c741.queue_time_us) (tablet-download-pool-48b0ff0769dc426190f038811155c741.run_cpu_time_us, tablet-download-pool-48b0ff0769dc426190f038811155c741.run_cpu_time_us) (tablet-download-pool-48b0ff0769dc426190f038811155c741.run_wall_time_us, tablet-download-pool-48b0ff0769dc426190f038811155c741.run_wall_time_us) (tablet-download-pool-5c47aaeb16c945ae84ece505aada2a7c.queue_time_us, tablet-download-pool-5c47aaeb16c945ae84ece505aada2a7c.queue_time_us) (tablet-download-pool-5c47aaeb16c945ae84ece505aada2a7c.run_cpu_time_us, tablet-download-pool-5c47aaeb16c945ae84ece505aada2a7c.run_cpu_time_us) (tablet-download-pool-5c47aaeb16c945ae84ece505aada2a7c.run_wall_time_us, tablet-download-pool-5c47aaeb16c945ae84ece505aada2a7c.run_wall_time_us) (tablet-download-pool-64613ea537c842f29d810e829278394e.queue_time_us, tablet-download-pool-64613ea537c842f29d810e829278394e.queue_time_us) (tablet-download-pool-64613ea537c842f29d810e829278394e.run_cpu_time_us, tablet-download-pool-64613ea537c842f29d810e829278394e.run_cpu_time_us) (tablet-download-pool-64613ea537c842f29d810e829278394e.run_wall_time_us, tablet-download-pool-64613ea537c842f29d810e829278394e.run_wall_time_us) (tablet-download-pool-6d214be86534458091f54a386d7d3334.queue_time_us, tablet-download-pool-6d214be86534458091f54a386d7d3334.queue_time_us) (tablet-download-pool-6d214be86534458091f54a386d7d3334.run_cpu_time_us, tablet-download-pool-6d214be86534458091f54a386d7d3334.run_cpu_time_us) (tablet-download-pool-6d214be86534458091f54a386d7d3334.run_wall_time_us, tablet-download-pool-6d214be86534458091f54a386d7d3334.run_wall_time_us) (tablet-download-pool-7c23e3bd7ec34876956046e5d32e4d82.queue_time_us, tablet-download-pool-7c23e3bd7ec34876956046e5d32e4d82.queue_time_us) (tablet-download-pool-7c23e3bd7ec34876956046e5d32e4d82.run_cpu_time_us, tablet-download-pool-7c23e3bd7ec34876956046e5d32e4d82.run_cpu_time_us) (tablet-download-pool-7c23e3bd7ec34876956046e5d32e4d82.run_wall_time_us, tablet-download-pool-7c23e3bd7ec34876956046e5d32e4d82.run_wall_time_us) (tablet-download-pool-867ba08a15b44d4c9e6fa92b0d518d1d.queue_time_us, tablet-download-pool-867ba08a15b44d4c9e6fa92b0d518d1d.queue_time_us) (tablet-download-pool-90f11b5693d7456d8a604e6c8726e69a.queue_time_us, tablet-download-pool-90f11b5693d7456d8a604e6c8726e69a.queue_time_us) (tablet-download-pool-90f11b5693d7456d8a604e6c8726e69a.run_cpu_time_us, tablet-download-pool-90f11b5693d7456d8a604e6c8726e69a.run_cpu_time_us) (tablet-download-pool-90f11b5693d7456d8a604e6c8726e69a.run_wall_time_us, tablet-download-pool-90f11b5693d7456d8a604e6c8726e69a.run_wall_time_us) (tablet-download-pool-a1ebf29682024645978b3caa2d4988f3.queue_time_us, tablet-download-pool-a1ebf29682024645978b3caa2d4988f3.queue_time_us) (tablet-download-pool-a1ebf29682024645978b3caa2d4988f3.run_cpu_time_us, tablet-download-pool-a1ebf29682024645978b3caa2d4988f3.run_cpu_time_us) (tablet-download-pool-a1ebf29682024645978b3caa2d4988f3.run_wall_time_us, tablet-download-pool-a1ebf29682024645978b3caa2d4988f3.run_wall_time_us) (tablet-download-pool-bd8348990f1b496bb62d8c0484193442.queue_time_us, tablet-download-pool-bd8348990f1b496bb62d8c0484193442.queue_time_us) (tablet-download-pool-bd8348990f1b496bb62d8c0484193442.run_cpu_time_us, tablet-download-pool-bd8348990f1b496bb62d8c0484193442.run_cpu_time_us) (tablet-download-pool-bd8348990f1b496bb62d8c0484193442.run_wall_time_us, tablet-download-pool-bd8348990f1b496bb62d8c0484193442.run_wall_time_us) (tablet-download-pool-ea97f34590684bcf998228245dbcd9b0.queue_time_us, tablet-download-pool-ea97f34590684bcf998228245dbcd9b0.queue_time_us) (tablet-download-pool-ea97f34590684bcf998228245dbcd9b0.run_cpu_time_us, tablet-download-pool-ea97f34590684bcf998228245dbcd9b0.run_cpu_time_us) (tablet-download-pool-ea97f34590684bcf998228245dbcd9b0.run_wall_time_us, tablet-download-pool-ea97f34590684bcf998228245dbcd9b0.run_wall_time_us) (tablet-download-pool-f2e63d24d7df4fa3b2b26695e70ebe9f.queue_time_us, tablet-download-pool-f2e63d24d7df4fa3b2b26695e70ebe9f.queue_time_us) (tablet-download-pool-f2e63d24d7df4fa3b2b26695e70ebe9f.run_cpu_time_us, tablet-download-pool-f2e63d24d7df4fa3b2b26695e70ebe9f.run_cpu_time_us) (tablet-download-pool-f2e63d24d7df4fa3b2b26695e70ebe9f.run_wall_time_us, tablet-download-pool-f2e63d24d7df4fa3b2b26695e70ebe9f.run_wall_time_us) (tablet-download-pool-f61e70ba02384f81b9ae36b1c65465b9.queue_time_us, tablet-download-pool-f61e70ba02384f81b9ae36b1c65465b9.queue_time_us) (tablet-download-pool-f61e70ba02384f81b9ae36b1c65465b9.run_cpu_time_us, tablet-download-pool-f61e70ba02384f81b9ae36b1c65465b9.run_cpu_time_us) (tablet-download-pool-f61e70ba02384f81b9ae36b1c65465b9.run_wall_time_us, tablet-download-pool-f61e70ba02384f81b9ae36b1c65465b9.run_wall_time_us) (tablet-download-pool-f80db2e37bbc4716ad3a5200c61ee97b.queue_time_us, tablet-download-pool-f80db2e37bbc4716ad3a5200c61ee97b.queue_time_us) (tablet-download-pool-f80db2e37bbc4716ad3a5200c61ee97b.run_cpu_time_us, tablet-download-pool-f80db2e37bbc4716ad3a5200c61ee97b.run_cpu_time_us) (tablet-download-pool-f80db2e37bbc4716ad3a5200c61ee97b.run_wall_time_us, tablet-download-pool-f80db2e37bbc4716ad3a5200c61ee97b.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fef2b69adcd google::LogMessage::Fail() at ??:0
@ 0x7fef2b69eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fef2b69a7cc google::LogMessage::Flush() at ??:0
@ 0x7fef2b69bf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fef2cb04044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fef2cae8f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fef2cae6efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fef2f69977a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fef2f6a2a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fef2f72376f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 08:00:10.458135 22586 ts_tablet_manager.cc:902] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: Initiating tablet copy from peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 08:00:10.459256 22586 tablet_copy_client.cc:308] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:44403
I20240501 08:00:10.460004 11683 tablet_copy_service.cc:140] P 603919bb02364c04b6c530245b8b02f2: Received BeginTabletCopySession request for tablet 5b7de08d2f004cd8abc2bbe1cf7b1444 from peer 22ec12774a4047a8b859743dc73f8ff6 ({username='slave'} at 127.0.99.131:37667)
I20240501 08:00:10.460117 11683 tablet_copy_service.cc:161] P 603919bb02364c04b6c530245b8b02f2: Beginning new tablet copy session on tablet 5b7de08d2f004cd8abc2bbe1cf7b1444 from peer 22ec12774a4047a8b859743dc73f8ff6 at {username='slave'} at 127.0.99.131:37667: session id = 22ec12774a4047a8b859743dc73f8ff6-5b7de08d2f004cd8abc2bbe1cf7b1444
@ 0x7fef2f7374f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 08:00:10.461103 11683 tablet_copy_source_session.cc:205] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:10.463351 22586 ts_tablet_manager.cc:1875] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:00:10.466274 22586 ts_tablet_manager.cc:1888] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 08:00:10.466565 22586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7de08d2f004cd8abc2bbe1cf7b1444. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.468086 22586 tablet_copy_client.cc:783] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:10.468264 22586 tablet_copy_client.cc:647] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: tablet copy: Starting download of 1 WAL segments...
@ 0x7fef2f722226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240501 08:00:10.472592 22586 tablet_copy_client.cc:523] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:10.474647 22586 tablet_bootstrap.cc:492] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
@ 0x7fef2f731c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x555668c9c75c std::function<>::operator()() at ??:0
I20240501 08:00:10.483099 22349 ts_tablet_manager.cc:902] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: Initiating tablet copy from peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 08:00:10.483605 22349 tablet_copy_client.cc:308] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37957
I20240501 08:00:10.484396 11551 tablet_copy_service.cc:140] P 22ec12774a4047a8b859743dc73f8ff6: Received BeginTabletCopySession request for tablet 5e8550ea95984d08937b9b3e2af12bec from peer 603919bb02364c04b6c530245b8b02f2 ({username='slave'} at 127.0.99.132:54785)
I20240501 08:00:10.484493 11551 tablet_copy_service.cc:161] P 22ec12774a4047a8b859743dc73f8ff6: Beginning new tablet copy session on tablet 5e8550ea95984d08937b9b3e2af12bec from peer 603919bb02364c04b6c530245b8b02f2 at {username='slave'} at 127.0.99.132:54785: session id = 603919bb02364c04b6c530245b8b02f2-5e8550ea95984d08937b9b3e2af12bec
I20240501 08:00:10.485585 11551 tablet_copy_source_session.cc:205] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:10.486369 22349 ts_tablet_manager.cc:1875] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:00:10.488787 22349 ts_tablet_manager.cc:1888] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 08:00:10.488968 22349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8550ea95984d08937b9b3e2af12bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.490566 22349 tablet_copy_client.cc:783] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:10.490705 22349 tablet_copy_client.cc:647] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: tablet copy: Starting download of 1 WAL segments...
@ 0x7fef2caebe06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fef2caec6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fef2caee03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x555668c9c75c std::function<>::operator()() at ??:0
I20240501 08:00:10.508589 22349 tablet_copy_client.cc:523] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:10.508798 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:10.510339 22349 tablet_bootstrap.cc:492] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
@ 0x7fef2cadb807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fef2d3196db start_thread at ??:0
W20240501 08:00:10.514382 13121 connection.cc:535] client connection to 127.0.99.133:42805 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:10.514508 11336 connection.cc:535] client connection to 127.0.99.133:42805 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:10.514568 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:10.514631 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:10.514703 11205 connection.cc:535] client connection to 127.0.99.133:42805 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:10.514829 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:10.515810 11467 connection.cc:535] client connection to 127.0.99.133:42805 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:10.515926 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:10.516474 11598 connection.cc:535] client connection to 127.0.99.133:42805 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:10.516577 11598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:00:10.533041 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
@ 0x7fef2a85171f clone at ??:0
I20240501 08:00:10.537720 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:10.538794 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:10.538915 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:10.538951 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:10.549145 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:10.551101 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:10.551187 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:10.551280 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:10.552731 22631 ts_tablet_manager.cc:902] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: Initiating tablet copy from peer 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771)
I20240501 08:00:10.553298 22631 tablet_copy_client.cc:308] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:36771
I20240501 08:00:10.553773 13206 tablet_copy_service.cc:140] P 53d1d2c0767b4f6195b69758a3dd6cf9: Received BeginTabletCopySession request for tablet 25cee7e51c7c438aa721d71fb82dc71e from peer e2dafc1866314669ad0de8fb7bf7fd05 ({username='slave'} at 127.0.99.129:58765)
I20240501 08:00:10.553886 13206 tablet_copy_service.cc:161] P 53d1d2c0767b4f6195b69758a3dd6cf9: Beginning new tablet copy session on tablet 25cee7e51c7c438aa721d71fb82dc71e from peer e2dafc1866314669ad0de8fb7bf7fd05 at {username='slave'} at 127.0.99.129:58765: session id = e2dafc1866314669ad0de8fb7bf7fd05-25cee7e51c7c438aa721d71fb82dc71e
I20240501 08:00:10.554957 13206 tablet_copy_source_session.cc:205] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:10.556622 22586 tablet_bootstrap.cc:492] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:10.556929 22586 tablet_bootstrap.cc:492] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap complete.
I20240501 08:00:10.557119 22586 ts_tablet_manager.cc:1367] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.083s user 0.027s sys 0.001s
I20240501 08:00:10.557626 22586 raft_consensus.cc:348] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 08:00:10.557745 22586 raft_consensus.cc:729] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
W20240501 08:00:10.560230 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:10.561694 22586 consensus_queue.cc:260] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 08:00:10.562179 22586 ts_tablet_manager.cc:1398] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:00:10.563833 11683 tablet_copy_service.cc:342] P 603919bb02364c04b6c530245b8b02f2: Request end of tablet copy session 22ec12774a4047a8b859743dc73f8ff6-5b7de08d2f004cd8abc2bbe1cf7b1444 received from {username='slave'} at 127.0.99.131:37667
I20240501 08:00:10.563923 11683 tablet_copy_service.cc:433] P 603919bb02364c04b6c530245b8b02f2: ending tablet copy session 22ec12774a4047a8b859743dc73f8ff6-5b7de08d2f004cd8abc2bbe1cf7b1444 on tablet 5b7de08d2f004cd8abc2bbe1cf7b1444 with peer 22ec12774a4047a8b859743dc73f8ff6
W20240501 08:00:10.566902 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:10.566977 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:10.567015 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:10.567049 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:10.567080 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:10.568773 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:10.581619 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:10.582813 22631 ts_tablet_manager.cc:1875] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240501 08:00:10.583065 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:10.586814 22631 ts_tablet_manager.cc:1888] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240501 08:00:10.587033 22631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25cee7e51c7c438aa721d71fb82dc71e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 08:00:10.594106 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:10.594125 22631 tablet_copy_client.cc:783] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:10.595212 22631 tablet_copy_client.cc:647] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:10.597074 22631 tablet_copy_client.cc:523] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:10.599772 22631 tablet_bootstrap.cc:492] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
W20240501 08:00:10.607671 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:10.609123 22349 tablet_bootstrap.cc:492] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:10.609498 22349 tablet_bootstrap.cc:492] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: Bootstrap complete.
I20240501 08:00:10.609691 22349 ts_tablet_manager.cc:1367] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.099s user 0.033s sys 0.000s
I20240501 08:00:10.610184 22349 raft_consensus.cc:348] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:00:10.610309 22349 raft_consensus.cc:729] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.610512 22349 consensus_queue.cc:260] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:00:10.610903 22349 ts_tablet_manager.cc:1398] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 08:00:10.613283 11551 tablet_copy_service.cc:342] P 22ec12774a4047a8b859743dc73f8ff6: Request end of tablet copy session 603919bb02364c04b6c530245b8b02f2-5e8550ea95984d08937b9b3e2af12bec received from {username='slave'} at 127.0.99.132:54785
I20240501 08:00:10.613396 11551 tablet_copy_service.cc:433] P 22ec12774a4047a8b859743dc73f8ff6: ending tablet copy session 603919bb02364c04b6c530245b8b02f2-5e8550ea95984d08937b9b3e2af12bec on tablet 5e8550ea95984d08937b9b3e2af12bec with peer 603919bb02364c04b6c530245b8b02f2
I20240501 08:00:10.632980 22631 tablet_bootstrap.cc:492] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:10.633216 22631 tablet_bootstrap.cc:492] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap complete.
I20240501 08:00:10.633359 22631 ts_tablet_manager.cc:1367] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.001s
I20240501 08:00:10.633733 22631 raft_consensus.cc:348] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 08:00:10.633836 22631 raft_consensus.cc:729] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.633957 22631 consensus_queue.cc:260] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 08:00:10.635143 22631 ts_tablet_manager.cc:1398] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 08:00:10.635620 13206 tablet_copy_service.cc:342] P 53d1d2c0767b4f6195b69758a3dd6cf9: Request end of tablet copy session e2dafc1866314669ad0de8fb7bf7fd05-25cee7e51c7c438aa721d71fb82dc71e received from {username='slave'} at 127.0.99.129:58765
I20240501 08:00:10.635699 13206 tablet_copy_service.cc:433] P 53d1d2c0767b4f6195b69758a3dd6cf9: ending tablet copy session e2dafc1866314669ad0de8fb7bf7fd05-25cee7e51c7c438aa721d71fb82dc71e on tablet 25cee7e51c7c438aa721d71fb82dc71e with peer e2dafc1866314669ad0de8fb7bf7fd05
I20240501 08:00:10.640033 13446 consensus_queue.cc:237] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:10.641266 13186 raft_consensus.cc:1261] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Refusing update from remote peer ca177138f2c44d2eafa1646b795f7608: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 08:00:10.641747 22348 consensus_queue.cc:1035] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:10.644062 22348 raft_consensus.cc:2932] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 9 to 10, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:10.645982 13186 raft_consensus.cc:2932] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 9 to 10, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:10.648834 11124 catalog_manager.cc:5484] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 reported cstate change: config changed from index 9 to 10, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New cstate: current_term: 3 leader_uuid: "ca177138f2c44d2eafa1646b795f7608" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:10.656069 11248 tablet_service.cc:1514] Processing DeleteTablet for tablet 53193305f868433c852755e8a8a81766 with delete_type TABLET_DATA_TOMBSTONED (TS e2dafc1866314669ad0de8fb7bf7fd05 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:42432
I20240501 08:00:10.656440 22680 tablet_replica.cc:330] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: stopping tablet replica
I20240501 08:00:10.656584 22680 raft_consensus.cc:2229] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:10.656752 22680 raft_consensus.cc:2258] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:10.657047 13446 consensus_queue.cc:237] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 08:00:10.657274 22680 ts_tablet_manager.cc:1875] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:10.659210 13186 raft_consensus.cc:1261] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Refusing update from remote peer ca177138f2c44d2eafa1646b795f7608: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 08:00:10.659698 22680 ts_tablet_manager.cc:1888] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.9
I20240501 08:00:10.659669 22348 consensus_queue.cc:1035] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:00:10.659757 22680 log.cc:1196] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/53193305f868433c852755e8a8a81766
W20240501 08:00:10.660285 13383 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085): Couldn't send request to peer e2dafc1866314669ad0de8fb7bf7fd05. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:10.660271 11113 catalog_manager.cc:4830] TS e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085): tablet 53193305f868433c852755e8a8a81766 (table rebalance_test_table_7 [id=36e20074d8dc4402bd7d9b2039213af5]) successfully deleted
I20240501 08:00:10.661895 22348 raft_consensus.cc:2932] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:00:10.664040 11115 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 53193305f868433c852755e8a8a81766 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:00:10.664849 13186 raft_consensus.cc:2932] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:00:10.665140 11124 catalog_manager.cc:5484] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 reported cstate change: config changed from index 10 to 11, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) added. New cstate: current_term: 3 leader_uuid: "ca177138f2c44d2eafa1646b795f7608" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:00:10.672413 11661 consensus_queue.cc:237] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:10.674306 13186 raft_consensus.cc:1261] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:00:10.674783 22275 consensus_queue.cc:1035] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:00:10.678898 22689 raft_consensus.cc:2932] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:10.697963 13186 raft_consensus.cc:2932] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:10.701347 11127 catalog_manager.cc:5484] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9 reported cstate change: config changed from index 6 to 7, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:10.704294 11510 tablet_service.cc:1514] Processing DeleteTablet for tablet f61e70ba02384f81b9ae36b1c65465b9 with delete_type TABLET_DATA_TOMBSTONED (TS 22ec12774a4047a8b859743dc73f8ff6 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52810
I20240501 08:00:10.714573 22701 tablet_replica.cc:330] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6: stopping tablet replica
I20240501 08:00:10.714740 22701 raft_consensus.cc:2229] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:10.714963 22701 raft_consensus.cc:2258] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:10.716117 22701 ts_tablet_manager.cc:1875] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:10.721037 22701 ts_tablet_manager.cc:1888] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:00:10.721130 22701 log.cc:1196] T f61e70ba02384f81b9ae36b1c65465b9 P 22ec12774a4047a8b859743dc73f8ff6: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/f61e70ba02384f81b9ae36b1c65465b9
I20240501 08:00:10.721706 11114 catalog_manager.cc:4830] TS 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957): tablet f61e70ba02384f81b9ae36b1c65465b9 (table rebalance_test_table_5 [id=0af21d4c4fcb49a7ac0756a5ded5d57d]) successfully deleted
I20240501 08:00:10.789423 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:10.794971 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:10.800144 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:10.803745 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:10.807636 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:10.813863 22586 ts_tablet_manager.cc:902] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: Initiating tablet copy from peer 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 08:00:10.814450 22586 tablet_copy_client.cc:308] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41575
I20240501 08:00:10.815017 11420 tablet_copy_service.cc:140] P 655ebda1fb2c49c4af7087a760529835: Received BeginTabletCopySession request for tablet 164fd265561e47128e7ff585b8cfab07 from peer 22ec12774a4047a8b859743dc73f8ff6 ({username='slave'} at 127.0.99.131:37679)
I20240501 08:00:10.815140 11420 tablet_copy_service.cc:161] P 655ebda1fb2c49c4af7087a760529835: Beginning new tablet copy session on tablet 164fd265561e47128e7ff585b8cfab07 from peer 22ec12774a4047a8b859743dc73f8ff6 at {username='slave'} at 127.0.99.131:37679: session id = 22ec12774a4047a8b859743dc73f8ff6-164fd265561e47128e7ff585b8cfab07
I20240501 08:00:10.816052 11420 tablet_copy_source_session.cc:205] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:10.816860 22586 ts_tablet_manager.cc:1875] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:00:10.819216 22586 ts_tablet_manager.cc:1888] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 08:00:10.819391 22586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164fd265561e47128e7ff585b8cfab07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:10.820545 22586 tablet_copy_client.cc:783] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:10.820676 22586 tablet_copy_client.cc:647] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:10.822896 22586 tablet_copy_client.cc:523] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:10.824640 22586 tablet_bootstrap.cc:492] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap starting.
I20240501 08:00:10.856866 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:10.858763 11468 connection.cc:535] client connection to 127.0.99.135:43995 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:10.860013 11599 connection.cc:535] client connection to 127.0.99.135:43995 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:10.860849 11206 connection.cc:535] client connection to 127.0.99.135:43995 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:10.862296 11337 connection.cc:535] client connection to 127.0.99.135:43995 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:10.863515 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:10.863579 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:10.863615 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:10.874828 22638 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:00:10.859415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:43995 (local address 127.0.99.131:51051)
0501 08:00:10.859601 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0501 08:00:10.859615 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 08:00:10.859696 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:00:10.874060 (+ 14364us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":110}
W20240501 08:00:10.875355 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:10.875432 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:10.875469 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:10.876570 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:10.877377 22632 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:00:10.860611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:43995 (local address 127.0.99.132:50747)
0501 08:00:10.860748 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0501 08:00:10.860763 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 08:00:10.860859 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:00:10.876812 (+ 15953us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":48}
W20240501 08:00:10.877554 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:10.877636 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:00:10.878300 22646 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:00:10.861591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:43995 (local address 127.0.99.129:50367)
0501 08:00:10.861781 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0501 08:00:10.861793 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 08:00:10.861876 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:00:10.877868 (+ 15992us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":113}
W20240501 08:00:10.878499 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:00:10.878559 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:10.879309 11661 raft_consensus.cc:1203] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240501 08:00:10.883131 22586 tablet_bootstrap.cc:492] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:00:10.883512 22586 tablet_bootstrap.cc:492] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: Bootstrap complete.
I20240501 08:00:10.883708 22586 ts_tablet_manager.cc:1367] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: Time spent bootstrapping tablet: real 0.059s user 0.024s sys 0.000s
I20240501 08:00:10.884169 22586 raft_consensus.cc:348] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:00:10.884692 22586 raft_consensus.cc:729] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ec12774a4047a8b859743dc73f8ff6, State: Initialized, Role: FOLLOWER
I20240501 08:00:10.884819 22586 consensus_queue.cc:260] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 08:00:10.885190 22586 ts_tablet_manager.cc:1398] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 08:00:10.885696 11420 tablet_copy_service.cc:342] P 655ebda1fb2c49c4af7087a760529835: Request end of tablet copy session 22ec12774a4047a8b859743dc73f8ff6-164fd265561e47128e7ff585b8cfab07 received from {username='slave'} at 127.0.99.131:37679
I20240501 08:00:10.885775 11420 tablet_copy_service.cc:433] P 655ebda1fb2c49c4af7087a760529835: ending tablet copy session 22ec12774a4047a8b859743dc73f8ff6-164fd265561e47128e7ff585b8cfab07 on tablet 164fd265561e47128e7ff585b8cfab07 with peer 22ec12774a4047a8b859743dc73f8ff6
I20240501 08:00:10.907940 13498 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 07:59:53.862679 13504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:53.862978 13504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:53.867719 13504 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 07:59:54.897323 13504 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 07:59:55.363677 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:55.363894 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df5f9864bbf8499dbaa392bcb85701c0: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:55.363998 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6447707eb34f45d7a8d3704cd09d7786: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:55.364051 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 36e20074d8dc4402bd7d9b2039213af5: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:55.364104 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 69f109081c7449be8819974c5363fa32: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:55.364197 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 701e50fb54354219a26221f76a7352d2: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:55.364255 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af21d4c4fcb49a7ac0756a5ded5d57d: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:55.364343 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24b4f6e8cc8c44f6961591cac7beacb4: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:55.364399 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 384b2b245c6b4931a4fc02c6b5d647b2: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:55.364492 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:55.364569 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af21d4c4fcb49a7ac0756a5ded5d57d: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:55.364611 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 98fb02bdb5f54b1699749e3c8d588a5e: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:55.364655 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df5f9864bbf8499dbaa392bcb85701c0: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:55.364729 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6447707eb34f45d7a8d3704cd09d7786: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:55.364815 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 69f109081c7449be8819974c5363fa32: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server ca177138f2c44d2eafa1646b795f7608
I20240501 07:59:55.586752 13504 rebalancer_tool.cc:1084] tablet f2e63d24d7df4fa3b2b26695e70ebe9f: 'e2dafc1866314669ad0de8fb7bf7fd05' -> 'e5c03261d31c4d9a91077a17e1cd0336' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 07:59:55.586920 13504 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 07:59:56.084390 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 69f109081c7449be8819974c5363fa32: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:56.084621 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 98fb02bdb5f54b1699749e3c8d588a5e: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:56.085490 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 69f109081c7449be8819974c5363fa32: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:56.085601 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6447707eb34f45d7a8d3704cd09d7786: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:56.085686 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 98fb02bdb5f54b1699749e3c8d588a5e: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:56.085753 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:56.085811 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24b4f6e8cc8c44f6961591cac7beacb4: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:56.085894 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df5f9864bbf8499dbaa392bcb85701c0: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:56.085950 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 701e50fb54354219a26221f76a7352d2: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:56.085995 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af21d4c4fcb49a7ac0756a5ded5d57d: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:56.086040 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:56.086099 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 36e20074d8dc4402bd7d9b2039213af5: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:56.086146 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 384b2b245c6b4931a4fc02c6b5d647b2: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:56.086237 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 69f109081c7449be8819974c5363fa32: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:56.086290 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 98fb02bdb5f54b1699749e3c8d588a5e: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:56.086341 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 701e50fb54354219a26221f76a7352d2: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:56.086392 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af21d4c4fcb49a7ac0756a5ded5d57d: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:56.086480 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server ca177138f2c44d2eafa1646b795f7608
I20240501 07:59:56.221760 13504 rebalancer_tool.cc:1074] tablet 164fd265561e47128e7ff585b8cfab07: '22ec12774a4047a8b859743dc73f8ff6' -> '60e8857abe7c41038af46ebb5a095a90' move scheduled
I20240501 07:59:56.276993 13504 rebalancer_tool.cc:1084] tablet b9431cf5853c4148ac38855b1c083d4f: '655ebda1fb2c49c4af7087a760529835' -> 'ca177138f2c44d2eafa1646b795f7608' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 07:59:56.277066 13504 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 07:59:56.879719 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 701e50fb54354219a26221f76a7352d2: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:56.879971 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af21d4c4fcb49a7ac0756a5ded5d57d: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:56.880036 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:56.880101 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 701e50fb54354219a26221f76a7352d2: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:56.880182 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 384b2b245c6b4931a4fc02c6b5d647b2: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:56.880254 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df5f9864bbf8499dbaa392bcb85701c0: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:56.880299 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 36e20074d8dc4402bd7d9b2039213af5: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:56.880342 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 69f109081c7449be8819974c5363fa32: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:56.880388 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 98fb02bdb5f54b1699749e3c8d588a5e: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:56.880437 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24b4f6e8cc8c44f6961591cac7beacb4: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:56.880507 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6447707eb34f45d7a8d3704cd09d7786: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:56.880579 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 36e20074d8dc4402bd7d9b2039213af5: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:56.880625 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24b4f6e8cc8c44f6961591cac7beacb4: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:56.880671 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df5f9864bbf8499dbaa392bcb85701c0: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:56.880717 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af21d4c4fcb49a7ac0756a5ded5d57d: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:56.880764 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:56.880805 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 701e50fb54354219a26221f76a7352d2: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:56.880851 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 69f109081c7449be8819974c5363fa32: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:56.880894 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 98fb02bdb5f54b1699749e3c8d588a5e: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:56.880944 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6447707eb34f45d7a8d3704cd09d7786: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:56.880990 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 384b2b245c6b4931a4fc02c6b5d647b2: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:56.881033 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 36e20074d8dc4402bd7d9b2039213af5: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server ca177138f2c44d2eafa1646b795f7608
I20240501 07:59:56.980576 13504 rebalancer_tool.cc:1084] tablet a1ebf29682024645978b3caa2d4988f3: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '53d1d2c0767b4f6195b69758a3dd6cf9' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 07:59:56.980670 13504 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 07:59:57.418931 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 701e50fb54354219a26221f76a7352d2: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:57.419155 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df5f9864bbf8499dbaa392bcb85701c0: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:57.419214 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 384b2b245c6b4931a4fc02c6b5d647b2: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:57.419258 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 36e20074d8dc4402bd7d9b2039213af5: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:57.419287 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24b4f6e8cc8c44f6961591cac7beacb4: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:57.419370 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:57.419396 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 701e50fb54354219a26221f76a7352d2: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:57.419422 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df5f9864bbf8499dbaa392bcb85701c0: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:57.419449 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af21d4c4fcb49a7ac0756a5ded5d57d: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:57.419473 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 36e20074d8dc4402bd7d9b2039213af5: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:57.419499 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24b4f6e8cc8c44f6961591cac7beacb4: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:57.419526 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 69f109081c7449be8819974c5363fa32: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:57.419557 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 98fb02bdb5f54b1699749e3c8d588a5e: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server e5c03261d31c4d9a91077a17e1cd0336
I20240501 07:59:57.583617 13504 rebalancer_tool.cc:1074] tablet 18b146cc4e164e1d9f9f827217c9b3ab: '655ebda1fb2c49c4af7087a760529835' -> '' move scheduled
I20240501 07:59:57.684870 13504 rebalancer_tool.cc:1074] tablet 5c47aaeb16c945ae84ece505aada2a7c: 'e2dafc1866314669ad0de8fb7bf7fd05' -> 'ca177138f2c44d2eafa1646b795f7608' move scheduled
I20240501 07:59:57.743094 13504 rebalancer_tool.cc:1084] tablet 48b0ff0769dc426190f038811155c741: '603919bb02364c04b6c530245b8b02f2' -> '53d1d2c0767b4f6195b69758a3dd6cf9' move ignored: Illegal state: tablet 48b0ff0769dc426190f038811155c741: replica 53d1d2c0767b4f6195b69758a3dd6cf9 is already present
I20240501 07:59:57.743201 13504 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 07:59:58.205255 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6447707eb34f45d7a8d3704cd09d7786: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:58.205410 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 384b2b245c6b4931a4fc02c6b5d647b2: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:58.205518 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af21d4c4fcb49a7ac0756a5ded5d57d: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:58.205592 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df5f9864bbf8499dbaa392bcb85701c0: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:58.205657 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:58.205732 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6447707eb34f45d7a8d3704cd09d7786: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:58.205793 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24b4f6e8cc8c44f6961591cac7beacb4: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:58.205850 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 69f109081c7449be8819974c5363fa32: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 60e8857abe7c41038af46ebb5a095a90
I20240501 07:59:58.358062 13504 rebalancer_tool.cc:1084] tablet 7a266f266c3d4fd49fa8b325f1045d61: '603919bb02364c04b6c530245b8b02f2' -> '53d1d2c0767b4f6195b69758a3dd6cf9' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 07:59:58.358166 13504 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 07:59:58.953151 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df5f9864bbf8499dbaa392bcb85701c0: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:58.953260 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:58.953290 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 384b2b245c6b4931a4fc02c6b5d647b2: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:58.953359 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24b4f6e8cc8c44f6961591cac7beacb4: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:58.953398 13504 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6447707eb34f45d7a8d3704cd09d7786: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 60e8857abe7c41038af46ebb5a095a90
I20240501 07:59:59.042264 13504 rebalancer_tool.cc:1074] tablet 6d214be86534458091f54a386d7d3334: '603919bb02364c04b6c530245b8b02f2' -> '' move scheduled
I20240501 07:59:59.115214 13504 rebalancer_tool.cc:1074] tablet 7c23e3bd7ec34876956046e5d32e4d82: '22ec12774a4047a8b859743dc73f8ff6' -> '' move scheduled
I20240501 07:59:59.200696 13504 rebalancer_tool.cc:1074] tablet f80db2e37bbc4716ad3a5200c61ee97b: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '' move scheduled
I20240501 07:59:59.224598 13504 rebalancer_tool.cc:1074] tablet 93602521de174997bda6702d25aafce1: '655ebda1fb2c49c4af7087a760529835' -> '' move scheduled
I20240501 07:59:59.855310 13504 rebalancer_tool.cc:1123] tablet 93602521de174997bda6702d25aafce1: '655ebda1fb2c49c4af7087a760529835' -> '' move is abandoned: Not found: Server with UUID 655ebda1fb2c49c4af7087a760529835 not a member of the config. RaftConfig: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:01.211745 13504 rebalancer_tool.cc:1136] tablet 7c23e3bd7ec34876956046e5d32e4d82: '22ec12774a4047a8b859743dc73f8ff6' -> '' move completed: OK
I20240501 08:00:01.226300 13504 rebalancer_tool.cc:1136] tablet 6d214be86534458091f54a386d7d3334: '603919bb02364c04b6c530245b8b02f2' -> '' move completed: OK
I20240501 08:00:02.105258 13504 rebalancer_tool.cc:1136] tablet f80db2e37bbc4716ad3a5200c61ee97b: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '' move completed: OK
I20240501 08:00:02.139268 13504 rebalancer_tool.cc:1074] tablet bd8348990f1b496bb62d8c0484193442: '603919bb02364c04b6c530245b8b02f2' -> '' move scheduled
I20240501 08:00:02.167198 13504 rebalancer_tool.cc:1074] tablet 867ba08a15b44d4c9e6fa92b0d518d1d: '22ec12774a4047a8b859743dc73f8ff6' -> '' move scheduled
I20240501 08:00:02.255108 13504 rebalancer_tool.cc:1074] tablet 5b7de08d2f004cd8abc2bbe1cf7b1444: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '' move scheduled
I20240501 08:00:02.285528 13504 rebalancer_tool.cc:1136] tablet 5b7de08d2f004cd8abc2bbe1cf7b1444: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '' move completed: OK
I20240501 08:00:02.351830 13504 rebalancer_tool.cc:1136] tablet 867ba08a15b44d4c9e6fa92b0d518d1d: '22ec12774a4047a8b859743dc73f8ff6' -> '' move completed: OK
I20240501 08:00:02.959017 13504 rebalancer_tool.cc:1136] tablet 18b146cc4e164e1d9f9f827217c9b3ab: '655ebda1fb2c49c4af7087a760529835' -> '' move completed: OK
I20240501 08:00:03.015175 13504 rebalancer_tool.cc:1136] tablet 164fd265561e47128e7ff585b8cfab07: '22ec12774a4047a8b859743dc73f8ff6' -> '60e8857abe7c41038af46ebb5a095a90' move completed: OK
I20240501 08:00:03.041958 13504 rebalancer_tool.cc:1136] tablet 5c47aaeb16c945ae84ece505aada2a7c: 'e2dafc1866314669ad0de8fb7bf7fd05' -> 'ca177138f2c44d2eafa1646b795f7608' move completed: OK
I20240501 08:00:03.055146 13504 rebalancer_tool.cc:1074] tablet 64613ea537c842f29d810e829278394e: '22ec12774a4047a8b859743dc73f8ff6' -> '' move scheduled
I20240501 08:00:03.069754 13504 rebalancer_tool.cc:1074] tablet a1ebf29682024645978b3caa2d4988f3: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '' move scheduled
I20240501 08:00:03.166510 13504 rebalancer_tool.cc:1074] tablet 90f11b5693d7456d8a604e6c8726e69a: '603919bb02364c04b6c530245b8b02f2' -> '' move scheduled
I20240501 08:00:03.249022 13504 rebalancer_tool.cc:1136] tablet 90f11b5693d7456d8a604e6c8726e69a: '603919bb02364c04b6c530245b8b02f2' -> '' move completed: OK
I20240501 08:00:03.884865 13504 rebalancer_tool.cc:1136] tablet 64613ea537c8
W20240501 08:00:10.914366 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:10.921254 11268 raft_consensus.cc:1203] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20240501 08:00:10.925462 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:10.934060 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:10.934255 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:10.936324 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:10.938603 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:10.954654 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:10.956840 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:10.991462 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:10.993818 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:11.012182 11531 raft_consensus.cc:1203] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240501 08:00:11.013901 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:11.048027 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:11.048112 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:11.050238 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:11.055735 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:11.070079 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:11.073401 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:11.092327 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:11.101377 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:11.110541 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:11.110971 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:11.125146 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:11.126998 22631 ts_tablet_manager.cc:902] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: Initiating tablet copy from peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503)
I20240501 08:00:11.127506 22631 tablet_copy_client.cc:308] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:34503
I20240501 08:00:11.129137 13469 tablet_copy_service.cc:140] P ca177138f2c44d2eafa1646b795f7608: Received BeginTabletCopySession request for tablet 53193305f868433c852755e8a8a81766 from peer e2dafc1866314669ad0de8fb7bf7fd05 ({username='slave'} at 127.0.99.129:36501)
I20240501 08:00:11.129233 13469 tablet_copy_service.cc:161] P ca177138f2c44d2eafa1646b795f7608: Beginning new tablet copy session on tablet 53193305f868433c852755e8a8a81766 from peer e2dafc1866314669ad0de8fb7bf7fd05 at {username='slave'} at 127.0.99.129:36501: session id = e2dafc1866314669ad0de8fb7bf7fd05-53193305f868433c852755e8a8a81766
I20240501 08:00:11.130163 13469 tablet_copy_source_session.cc:205] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:11.130867 22631 ts_tablet_manager.cc:1875] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:00:11.133872 22631 ts_tablet_manager.cc:1888] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 3.9
I20240501 08:00:11.134053 22631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53193305f868433c852755e8a8a81766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.136281 22631 tablet_copy_client.cc:783] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:11.136441 22631 tablet_copy_client.cc:647] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: tablet copy: Starting download of 1 WAL segments...
I20240501 08:00:11.138160 22631 tablet_copy_client.cc:523] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:11.140017 22631 tablet_bootstrap.cc:492] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 08:00:11.142014 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:11.168339 22631 tablet_bootstrap.cc:492] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:11.168663 22631 tablet_bootstrap.cc:492] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap complete.
I20240501 08:00:11.168848 22631 ts_tablet_manager.cc:1367] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.029s user 0.025s sys 0.001s
I20240501 08:00:11.169337 22631 raft_consensus.cc:348] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 08:00:11.169461 22631 raft_consensus.cc:729] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 08:00:11.169606 22631 consensus_queue.cc:260] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 3.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 08:00:11.170459 22631 ts_tablet_manager.cc:1398] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:00:11.170940 13469 tablet_copy_service.cc:342] P ca177138f2c44d2eafa1646b795f7608: Request end of tablet copy session e2dafc1866314669ad0de8fb7bf7fd05-53193305f868433c852755e8a8a81766 received from {username='slave'} at 127.0.99.129:36501
I20240501 08:00:11.171029 13469 tablet_copy_service.cc:433] P ca177138f2c44d2eafa1646b795f7608: ending tablet copy session e2dafc1866314669ad0de8fb7bf7fd05-53193305f868433c852755e8a8a81766 on tablet 53193305f868433c852755e8a8a81766 with peer e2dafc1866314669ad0de8fb7bf7fd05
I20240501 08:00:11.210543 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:11.210952 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:11.215021 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:11.215484 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:11.215972 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:11.216495 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:11.298364 11531 raft_consensus.cc:1203] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 08:00:11.298722 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:11.300379 22821 raft_consensus.cc:2932] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 08:00:11.302533 11531 raft_consensus.cc:2932] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 08:00:11.302681 22821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:53192 (request call id 0) took 1033 ms. Trace:
I20240501 08:00:11.302757 22821 rpcz_store.cc:276] 0501 08:00:10.269643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 08:00:10.269695 (+ 52us) service_pool.cc:225] Handling call
0501 08:00:10.271011 (+ 1316us) log.cc:842] Serialized 482 byte log entry
0501 08:00:11.302663 (+1031652us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":51,"raft.queue_time_us":772,"raft.run_cpu_time_us":327,"raft.run_wall_time_us":326,"thread_start_us":81,"threads_started":1,"wal-append.queue_time_us":7181,"wal-append.run_cpu_time_us":469,"wal-append.run_wall_time_us":1000684}
I20240501 08:00:11.305560 13446 raft_consensus.cc:2932] T 164fd265561e47128e7ff585b8cfab07 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 08:00:11.311345 11127 catalog_manager.cc:5484] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 7 to 8, VOTER 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 08:00:11.316561 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:11.317384 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:11.318841 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:11.318878 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:11.327473 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:11.386325 11531 consensus_queue.cc:237] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 08:00:11.388015 11661 raft_consensus.cc:1261] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:00:11.388249 13446 raft_consensus.cc:1261] T 5e8550ea95984d08937b9b3e2af12bec P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
W20240501 08:00:11.388533 11465 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575): Couldn't send request to peer 655ebda1fb2c49c4af7087a760529835. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:11.388734 23021 consensus_queue.cc:1035] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 08:00:11.388895 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:11.389063 23021 consensus_queue.cc:1035] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:00:11.390830 23020 raft_consensus.cc:2932] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 08:00:11.391363 13446 raft_consensus.cc:2932] T 5e8550ea95984d08937b9b3e2af12bec P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 08:00:11.394093 11119 catalog_manager.cc:5484] T 5e8550ea95984d08937b9b3e2af12bec P ca177138f2c44d2eafa1646b795f7608 reported cstate change: config changed from index 9 to 10, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) added. New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 08:00:11.394165 11661 raft_consensus.cc:2932] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
W20240501 08:00:11.423642 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:11.466899 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:11.467942 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:11.473996 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:11.476150 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:11.476962 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:11.490772 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:11.492000 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:11.614838 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:11.620834 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:11.621759 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:11.623243 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:11.625728 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:11.640280 11268 raft_consensus.cc:1203] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.10->[3.11-3.11] Dedup: 3.11->[]
I20240501 08:00:11.646739 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:11.646828 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:11.648020 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:11.649029 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:11.672127 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:11.674244 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:11.679724 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:11.757622 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:11.757987 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:11.764659 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:11.774214 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:11.776167 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:11.777475 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:11.855821 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:11.855836 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:11.856830 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:11.862939 13446 consensus_queue.cc:237] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:00:11.864326 13186 raft_consensus.cc:1261] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Refusing update from remote peer ca177138f2c44d2eafa1646b795f7608: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240501 08:00:11.864750 22682 consensus_queue.cc:1035] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:00:11.864817 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:11.867125 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:11.878634 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:11.881183 13186 raft_consensus.cc:2932] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:11.880738 23258 raft_consensus.cc:2932] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:11.884171 11123 catalog_manager.cc:5484] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 reported cstate change: config changed from index 11 to 12, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) evicted. New cstate: current_term: 3 leader_uuid: "ca177138f2c44d2eafa1646b795f7608" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:00:11.887454 11248 tablet_service.cc:1514] Processing DeleteTablet for tablet 53193305f868433c852755e8a8a81766 with delete_type TABLET_DATA_TOMBSTONED (TS e2dafc1866314669ad0de8fb7bf7fd05 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:42432
I20240501 08:00:11.889668 13446 consensus_queue.cc:237] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:00:11.891721 13186 raft_consensus.cc:1261] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Refusing update from remote peer ca177138f2c44d2eafa1646b795f7608: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 08:00:11.892704 22682 consensus_queue.cc:1035] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240501 08:00:11.893739 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:00:11.895107 23259 tablet_replica.cc:330] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: stopping tablet replica
I20240501 08:00:11.895334 23259 raft_consensus.cc:2229] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:11.895485 23259 raft_consensus.cc:2258] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:11.896152 22682 raft_consensus.cc:2932] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:11.896512 23259 ts_tablet_manager.cc:1875] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:11.898654 11115 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 53193305f868433c852755e8a8a81766 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:00:11.899299 23259 ts_tablet_manager.cc:1888] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.11
I20240501 08:00:11.899463 23259 log.cc:1196] T 53193305f868433c852755e8a8a81766 P e2dafc1866314669ad0de8fb7bf7fd05: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/53193305f868433c852755e8a8a81766
I20240501 08:00:11.899477 13186 raft_consensus.cc:2932] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:11.901418 11113 catalog_manager.cc:4830] TS e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085): tablet 53193305f868433c852755e8a8a81766 (table rebalance_test_table_7 [id=36e20074d8dc4402bd7d9b2039213af5]) successfully deleted
I20240501 08:00:11.902281 11128 catalog_manager.cc:5484] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 reported cstate change: config changed from index 12 to 13, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) added. New cstate: current_term: 3 leader_uuid: "ca177138f2c44d2eafa1646b795f7608" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } }
I20240501 08:00:11.982228 23312 ts_tablet_manager.cc:902] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: Initiating tablet copy from peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 08:00:11.983151 23312 tablet_copy_client.cc:308] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37957
I20240501 08:00:11.984162 11551 tablet_copy_service.cc:140] P 22ec12774a4047a8b859743dc73f8ff6: Received BeginTabletCopySession request for tablet 5e8550ea95984d08937b9b3e2af12bec from peer 655ebda1fb2c49c4af7087a760529835 ({username='slave'} at 127.0.99.130:44689)
I20240501 08:00:11.984289 11551 tablet_copy_service.cc:161] P 22ec12774a4047a8b859743dc73f8ff6: Beginning new tablet copy session on tablet 5e8550ea95984d08937b9b3e2af12bec from peer 655ebda1fb2c49c4af7087a760529835 at {username='slave'} at 127.0.99.130:44689: session id = 655ebda1fb2c49c4af7087a760529835-5e8550ea95984d08937b9b3e2af12bec
I20240501 08:00:11.985420 11551 tablet_copy_source_session.cc:205] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:00:11.986828 23312 ts_tablet_manager.cc:1875] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:00:11.991848 23312 ts_tablet_manager.cc:1888] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240501 08:00:11.992184 23312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8550ea95984d08937b9b3e2af12bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:00:11.996433 23312 tablet_copy_client.cc:783] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: tablet copy: Starting download of 0 data blocks...
I20240501 08:00:11.996786 23312 tablet_copy_client.cc:647] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: tablet copy: Starting download of 1 WAL segments...
W20240501 08:00:11.998947 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:11.999719 23312 tablet_copy_client.cc:523] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:00:12.000195 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:12.001788 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:12.007979 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:12.009274 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:12.011102 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:12.022485 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:12.027868 23312 tablet_bootstrap.cc:492] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
W20240501 08:00:12.044229 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:12.071800 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:12.076654 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:12.078644 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:12.079823 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:12.092273 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:12.096283 23312 tablet_bootstrap.cc:492] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:00:12.096755 23312 tablet_bootstrap.cc:492] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: Bootstrap complete.
I20240501 08:00:12.096982 23312 ts_tablet_manager.cc:1367] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.069s user 0.035s sys 0.000s
I20240501 08:00:12.097505 23312 raft_consensus.cc:348] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 08:00:12.097656 23312 raft_consensus.cc:729] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 08:00:12.097926 23312 consensus_queue.cc:260] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 08:00:12.098260 23312 ts_tablet_manager.cc:1398] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:00:12.098953 11551 tablet_copy_service.cc:342] P 22ec12774a4047a8b859743dc73f8ff6: Request end of tablet copy session 655ebda1fb2c49c4af7087a760529835-5e8550ea95984d08937b9b3e2af12bec received from {username='slave'} at 127.0.99.130:44689
I20240501 08:00:12.099058 11551 tablet_copy_service.cc:433] P 22ec12774a4047a8b859743dc73f8ff6: ending tablet copy session 655ebda1fb2c49c4af7087a760529835-5e8550ea95984d08937b9b3e2af12bec on tablet 5e8550ea95984d08937b9b3e2af12bec with peer 655ebda1fb2c49c4af7087a760529835
I20240501 08:00:12.114012 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:12.239141 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:12.251315 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:12.256428 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:12.257300 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:12.257620 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:12.265095 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:12.265709 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:12.266849 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:12.267457 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:12.294543 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:12.301416 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:12.305706 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:12.356433 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:12.369387 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:12.375738 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:12.376485 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:12.388216 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:12.405494 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:12.411674 13502 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 07:59:53.867842 13505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:53.868386 13505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:53.873162 13505 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 07:59:54.993863 13505 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 07:59:55.321317 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af21d4c4fcb49a7ac0756a5ded5d57d: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:55.321467 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 701e50fb54354219a26221f76a7352d2: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:55.321602 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 36e20074d8dc4402bd7d9b2039213af5: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:55.321659 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 69f109081c7449be8819974c5363fa32: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:55.321696 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 98fb02bdb5f54b1699749e3c8d588a5e: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:55.321736 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:55.321772 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af21d4c4fcb49a7ac0756a5ded5d57d: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:55.321830 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24b4f6e8cc8c44f6961591cac7beacb4: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:55.321878 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6447707eb34f45d7a8d3704cd09d7786: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:55.321928 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 384b2b245c6b4931a4fc02c6b5d647b2: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:55.321974 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df5f9864bbf8499dbaa392bcb85701c0: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:55.322036 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 98fb02bdb5f54b1699749e3c8d588a5e: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:55.322081 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:55.322134 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 69f109081c7449be8819974c5363fa32: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server ca177138f2c44d2eafa1646b795f7608
I20240501 07:59:55.571085 13505 rebalancer_tool.cc:1074] tablet f2e63d24d7df4fa3b2b26695e70ebe9f: 'e2dafc1866314669ad0de8fb7bf7fd05' -> 'ca177138f2c44d2eafa1646b795f7608' move scheduled
I20240501 07:59:55.687634 13505 rebalancer_tool.cc:1074] tablet 93602521de174997bda6702d25aafce1: '655ebda1fb2c49c4af7087a760529835' -> '53d1d2c0767b4f6195b69758a3dd6cf9' move scheduled
I20240501 07:59:55.781597 13505 rebalancer_tool.cc:1074] tablet 64613ea537c842f29d810e829278394e: '22ec12774a4047a8b859743dc73f8ff6' -> 'e5c03261d31c4d9a91077a17e1cd0336' move scheduled
I20240501 07:59:56.100229 13505 rebalancer_tool.cc:1074] tablet 6d214be86534458091f54a386d7d3334: '603919bb02364c04b6c530245b8b02f2' -> '60e8857abe7c41038af46ebb5a095a90' move scheduled
I20240501 07:59:56.188539 13505 rebalancer_tool.cc:1074] tablet f80db2e37bbc4716ad3a5200c61ee97b: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '53d1d2c0767b4f6195b69758a3dd6cf9' move scheduled
I20240501 07:59:56.260686 13505 rebalancer_tool.cc:1074] tablet 867ba08a15b44d4c9e6fa92b0d518d1d: '22ec12774a4047a8b859743dc73f8ff6' -> 'ca177138f2c44d2eafa1646b795f7608' move scheduled
I20240501 07:59:56.380520 13505 rebalancer_tool.cc:1074] tablet f61e70ba02384f81b9ae36b1c65465b9: '655ebda1fb2c49c4af7087a760529835' -> 'e5c03261d31c4d9a91077a17e1cd0336' move scheduled
I20240501 07:59:56.705471 13505 rebalancer_tool.cc:1074] tablet 25cee7e51c7c438aa721d71fb82dc71e: '603919bb02364c04b6c530245b8b02f2' -> '60e8857abe7c41038af46ebb5a095a90' move scheduled
I20240501 07:59:56.938759 13505 rebalancer_tool.cc:1074] tablet b4d15e19eed54aefb6b96276fad98324: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '53d1d2c0767b4f6195b69758a3dd6cf9' move scheduled
I20240501 07:59:57.007416 13505 rebalancer_tool.cc:1074] tablet 164fd265561e47128e7ff585b8cfab07: '22ec12774a4047a8b859743dc73f8ff6' -> 'ca177138f2c44d2eafa1646b795f7608' move scheduled
I20240501 07:59:57.233759 13505 rebalancer_tool.cc:1084] tablet 53193305f868433c852755e8a8a81766: '655ebda1fb2c49c4af7087a760529835' -> 'e5c03261d31c4d9a91077a17e1cd0336' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 07:59:57.233898 13505 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 07:59:57.634071 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6447707eb34f45d7a8d3704cd09d7786: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:57.634186 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 384b2b245c6b4931a4fc02c6b5d647b2: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:57.634265 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:57.634318 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24b4f6e8cc8c44f6961591cac7beacb4: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:57.634390 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df5f9864bbf8499dbaa392bcb85701c0: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:57.634533 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 69f109081c7449be8819974c5363fa32: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:57.634614 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 701e50fb54354219a26221f76a7352d2: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:57.634663 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af21d4c4fcb49a7ac0756a5ded5d57d: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:57.634708 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 36e20074d8dc4402bd7d9b2039213af5: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:57.634753 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:57.634793 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24b4f6e8cc8c44f6961591cac7beacb4: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:57.634833 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6447707eb34f45d7a8d3704cd09d7786: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:57.634865 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df5f9864bbf8499dbaa392bcb85701c0: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:57.634907 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 98fb02bdb5f54b1699749e3c8d588a5e: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server ca177138f2c44d2eafa1646b795f7608
I20240501 07:59:57.692600 13505 rebalancer_tool.cc:1074] tablet 18b146cc4e164e1d9f9f827217c9b3ab: '655ebda1fb2c49c4af7087a760529835' -> '' move scheduled
I20240501 07:59:57.745652 13505 rebalancer_tool.cc:1074] tablet 90f11b5693d7456d8a604e6c8726e69a: '603919bb02364c04b6c530245b8b02f2' -> '' move scheduled
I20240501 07:59:57.942173 13505 rebalancer_tool.cc:1074] tablet 5c47aaeb16c945ae84ece505aada2a7c: '603919bb02364c04b6c530245b8b02f2' -> 'e5c03261d31c4d9a91077a17e1cd0336' move scheduled
I20240501 07:59:58.064888 13505 rebalancer_tool.cc:1074] tablet b4707b952d174a9bb877ad8bd7a11268: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '60e8857abe7c41038af46ebb5a095a90' move scheduled
I20240501 07:59:58.088174 13505 rebalancer_tool.cc:1074] tablet 5e8550ea95984d08937b9b3e2af12bec: '655ebda1fb2c49c4af7087a760529835' -> '' move scheduled
I20240501 07:59:58.236573 13505 rebalancer_tool.cc:1074] tablet a24be92aec7544089c7818f2802b9c85: 'e2dafc1866314669ad0de8fb7bf7fd05' -> 'ca177138f2c44d2eafa1646b795f7608' move scheduled
I20240501 07:59:58.338165 13505 rebalancer_tool.cc:1084] tablet 7a266f266c3d4fd49fa8b325f1045d61: '603919bb02364c04b6c530245b8b02f2' -> 'e5c03261d31c4d9a91077a17e1cd0336' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 07:59:58.338279 13505 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 07:59:58.964663 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 36e20074d8dc4402bd7d9b2039213af5: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:58.978950 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 384b2b245c6b4931a4fc02c6b5d647b2: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:58.979099 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6447707eb34f45d7a8d3704cd09d7786: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:58.983587 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df5f9864bbf8499dbaa392bcb85701c0: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:58.983752 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 69f109081c7449be8819974c5363fa32: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:58.983814 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af21d4c4fcb49a7ac0756a5ded5d57d: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:58.983870 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:58.983925 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 701e50fb54354219a26221f76a7352d2: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:58.984009 13505 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 384b2b245c6b4931a4fc02c6b5d647b2: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server ca177138f2c44d2eafa1646b795f7608
I20240501 07:59:59.129973 13505 rebalancer_tool.cc:1074] tablet 7c23e3bd7ec34876956046e5d32e4d82: '22ec12774a4047a8b859743dc73f8ff6' -> '' move scheduled
I20240501 07:59:59.210492 13505 rebalancer_tool.cc:1074] tablet 48b0ff0769dc426190f038811155c741: '655ebda1fb2c49c4af7087a760529835' -> '' move scheduled
I20240501 08:00:00.459893 13505 rebalancer_tool.cc:1136] tablet f80db2e37bbc4716ad3a5200c61ee97b: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '53d1d2c0767b4f6195b69758a3dd6cf9' move completed: OK
I20240501 08:00:01.664878 13505 rebalancer_tool.cc:1136] tablet 867ba08a15b44d4c9e6fa92b0d518d1d: '22ec12774a4047a8b859743dc73f8ff6' -> 'ca177138f2c44d2eafa1646b795f7608' move completed: OK
I20240501 08:00:03.394874 13505 rebalancer_tool.cc:1136] tablet 6d214be86534458091f54a386d7d3334: '603919bb02364c04b6c530245b8b02f2' -> '60e8857abe7c41038af46ebb5a095a90' move completed: OK
I20240501 08:00:03.421236 13505 rebalancer_tool.cc:1136] tablet 93602521de174997bda6702d25aafce1: '655ebda1fb2c49c4af7087a760529835' -> '53d1d2c0767b4f6195b69758a3dd6cf9' move completed: OK
I20240501 08:00:03.440037 13505 rebalancer_tool.cc:1136] tablet f61e70ba02384f81b9ae36b1c65465b9: '655ebda1fb2c49c4af7087a760529835' -> 'e5c03261d31c4d9a91077a17e1cd0336' move completed: OK
I20240501 08:00:03.983981 13505 rebalancer_tool.cc:1136] tablet 18b146cc4e164e1d9f9f827217c9b3ab: '655ebda1fb2c49c4af7087a760529835' -> '' move completed: OK
I20240501 08:00:06.512498 13505 rebalancer_tool.cc:1136] tablet 164fd265561e47128e7ff585b8cfab07: '22ec12774a4047a8b859743dc73f8ff6' -> 'ca177138f2c44d2eafa1646b795f7608' move completed: OK
I20240501 08:00:06.597999 13505 rebalancer_tool.cc:1136] tablet b4707b952d174a9bb877ad8bd7a11268: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '60e8857abe7c41038af46ebb5a095a90' move completed: OK
I20240501 08:00:06.611927 13505 rebalancer_tool.cc:1074] tablet c95c4f3ef08f44d1b69fb7b98426f5af: '655ebda1fb2c49c4af7087a760529835' -> '' move scheduled
I20240501 08:00:06.629105 13505 rebalancer_tool.cc:1136] tablet c95c4f3ef08f44d1b69fb7b98426f5af: '655ebda1fb2c49c4af7087a760529835' -> '' move completed: OK
I20240501 08:00:06.644309 13505 rebalancer_tool.cc:1136] tablet 48b0ff0769dc426190f038811155c741: '655ebda1fb2c49c4af7087a760529835' -> '' move completed: OK
I20240501 08:00:06.659409 13505 rebalancer_tool.cc:1136] tablet 7c23e3bd7ec34876956046e5d32e4d82: '22ec12774a4047a8b859743dc73f8ff6' -> '' move completed: OK
I20240501 08:00:06.705065 13505 rebalancer_tool.cc:1136] tablet 5c47aaeb16c945ae84ece505aada2a7c: '603919bb02364c04b6c530245b8b02f2' -> 'e5c03261d31c4d9a91077a17e1cd0336' move completed: OK
I20240501 08:00:06.730947 13505 rebalancer_tool.cc:1136] tablet 64613ea537c842f29d810e829278394e: '22ec12774a4047a8b859743dc73f8ff6' -> 'e5c03261d31c4d9a91077a17e1cd0336' move completed: OK
I20240501 08:00:07.282492 13505 rebalancer_tool.cc:1136] tablet a24be92aec7544089c7818f2802b9c85: 'e2dafc1866314669ad0de8fb7bf7fd05' -> 'ca177138f2c44d2eafa1646b795f7608' move completed: OK
I20240501 08:00:07.852097 13505 rebalancer_tool.cc:1136] tablet b4d15e19eed54aefb6b96276fad98324: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '53d1d2c0767b4f6195b69758a3dd6cf9' move completed: OK
I20240501 08:00:07.867168 13505 rebalancer_tool.cc:1136] tablet f2e63d24d7df4fa3b2b26695e70ebe9f: 'e2dafc1866314669ad0de8fb7bf7fd05' -> 'ca177138f2c44d2eafa1646b795f7608' move completed: OK
I20240501 08:00:08.311079 13505 rebalancer_tool.cc:1074] tablet b9431cf5853c4148ac38855b1c083d4f: '22ec12774a4047a8b859743dc73f8ff6' -> '' move scheduled
I20240501 08:00:08.342797 13505 rebalancer_tool.cc:1074] tablet 5b7de08d2f004cd8abc2bbe1cf7b1444: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '' move scheduled
I20240501 08:00:08.354642 13505 rebalancer_tool.cc:1074] tablet 53193305f868433c852755e8a8a81766: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '' move scheduled
I20240501 08:00:09.005255 13505 rebalancer_tool.cc:1136] tablet 5b7de08d2f004cd8abc2bbe1cf7b1444: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '' move completed: OK
I20240501 08:00:09.038324 13505 rebalancer_tool.cc:1136] tablet b9431cf5853c4148ac38855b1c083d4f: '22ec12774a4047a8b859743dc73f8ff6' -> '' move completed: OK
I20240501 08:00:09.048317 13505 rebalancer_tool.cc:1136] tablet 90f11b5693d7456d8a604e6c8726e69a: '603919bb02364c04b6c530245b8b02f2' -> '' move completed: OK
I20240501 08:00:09.061764 13505 rebalancer_tool.cc:1136] tablet 25cee7e51c7c438aa721d71fb82dc71e: '603919bb02364c04b6c530245b8b02f2' -> '60e8857abe7c41038af46ebb5a095a90' move completed: OK
I20240501 08:00:09.085448 13505 rebalancer_tool.cc:1136] tablet 5e8550ea95984d08937b9b3e2af12bec: '655ebda1fb2c49c4af7087a760529835' -> '' move completed: OK
I20240501 08:00:09.097348 13505 rebalancer_tool.cc:1074] tablet a1ebf29682024645978b3caa2d4988f3: '603919bb02364c04b6c530245b8b02f2' -> '' move scheduled
W20240501 08:00:10.516980 22520 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:00:10.170019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:42805 (local address 127.0.0.1:60342)
0501 08:00:10.171637 (+ 1618us) negotiation.cc:107] Waiting for socket to connect
0501 08:00:10.171650 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 08:00:10.171719 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:00:10.516409 (+344690us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1555,"thread_start_us":101,"threads_started":1}
I20240501 08:00:10.893803 13505 rebalancer_tool.cc:1136] tablet a1ebf29682024645978b3caa2d4988f3: '603919bb02364c04b6c530245b8b02f2' -> '' move completed: OK
I20240501 08:00:12.104488 13505 rebalancer_tool.cc:1136] tablet 53193305f868433c852755e8a8a81766: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '' move completed: OK
Illegal state: tablet server 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): unacceptable health status UNAVAILABLE
W20240501 08:00:12.439922 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:12.499502 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:00:12.501956 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:12.508096 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:12.516497 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:12.520961 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:12.522289 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:12.529210 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:12.537417 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:12.538600 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:12.539552 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:12.540956 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:12.569372 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:12.570091 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:12.570461 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:12.597502 11401 raft_consensus.cc:1203] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
W20240501 08:00:12.611935 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:12.640494 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:12.648350 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:12.648685 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:12.653808 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:12.658643 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:12.673589 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:12.686748 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:12.704728 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:12.725577 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:00:12.746068 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:12.746769 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:12.748160 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:12.752866 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:00:12.769389 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:12.769662 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:12.772894 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:12.785701 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:12.807379 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:00:12.834311 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:12.835256 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:12.837558 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:12.839413 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:12.844695 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:12.846896 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:12.848037 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:12.868079 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:12.891433 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:12.920187 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:00:12.929864 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:12.931533 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:00:12.932963 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:12.936671 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:12.944237 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:00:12.944350 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:12.953315 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:12.959882 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:13.012378 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:13.020622 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:13.023275 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:13.023454 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:13.032429 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:13.033015 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:13.099915 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:13.109428 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:13.111480 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:13.111588 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:13.115267 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:13.117229 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:13.120584 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:13.142553 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:13.143004 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:13.173998 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:13.193852 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:13.202126 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:00:13.204387 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:13.206494 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:13.206513 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:13.211103 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:13.217725 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:13.218449 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:00:13.219496 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:13.243723 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:13.252842 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:13.271687 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:00:13.295742 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:13.304771 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:13.319072 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:13.323010 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:13.326077 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:13.335139 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:13.403676 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:13.412297 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:13.414619 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:13.426573 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:00:13.430491 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:13.435746 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:13.449270 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:13.454816 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:13.465178 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:13.467897 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:00:13.469028 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:13.469403 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:13.474800 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:13.479775 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:13.485414 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:13.503791 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:00:13.548491 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:00:13.555079 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:13.558225 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:13.559753 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:13.570678 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:13.572969 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:13.573804 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:13.606566 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:13.672695 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:13.675230 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:13.685978 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:13.686820 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:13.691144 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:13.714802 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:13.723040 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:13.815815 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:13.832065 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:13.840698 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:13.842283 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:13.848678 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:13.853721 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:13.854583 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:13.856873 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:13.859911 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:13.862411 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:13.897737 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:13.924796 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:13.936498 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:13.942049 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:13.947410 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:13.948828 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:13.965335 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:13.969599 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:14.026311 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:00:14.062026 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:14.064209 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:14.069916 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:14.083906 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:14.086944 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:14.091708 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:14.105490 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:14.108016 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:14.111461 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:14.114614 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:14.126458 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:14.133981 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:14.160602 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:14.199216 11662 consensus_queue.cc:237] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 08:00:14.200706 13185 raft_consensus.cc:1261] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:14.201200 24043 consensus_queue.cc:1035] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:14.201301 11532 raft_consensus.cc:1261] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:00:14.202958 24346 consensus_queue.cc:1035] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:00:14.204985 24043 raft_consensus.cc:2932] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 08:00:14.206051 11532 raft_consensus.cc:2932] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 08:00:14.208695 11122 catalog_manager.cc:5484] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 8 to 9, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 08:00:14.208958 13186 raft_consensus.cc:2932] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 08:00:14.216037 13429 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b7de08d2f004cd8abc2bbe1cf7b1444 with delete_type TABLET_DATA_TOMBSTONED (TS ca177138f2c44d2eafa1646b795f7608 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:42482
I20240501 08:00:14.216989 24373 tablet_replica.cc:330] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608: stopping tablet replica
I20240501 08:00:14.217329 24373 raft_consensus.cc:2229] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:00:14.217689 24373 raft_consensus.cc:2258] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:00:14.219000 24373 ts_tablet_manager.cc:1875] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:00:14.221925 24373 ts_tablet_manager.cc:1888] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:00:14.222776 24373 log.cc:1196] T 5b7de08d2f004cd8abc2bbe1cf7b1444 P ca177138f2c44d2eafa1646b795f7608: Deleting WAL directory at /tmp/dist-test-taskvJ2Dnc/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714550306649372-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/5b7de08d2f004cd8abc2bbe1cf7b1444
I20240501 08:00:14.223361 11115 catalog_manager.cc:4830] TS ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): tablet 5b7de08d2f004cd8abc2bbe1cf7b1444 (table rebalance_test_table_6 [id=701e50fb54354219a26221f76a7352d2]) successfully deleted
I20240501 08:00:14.258775 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:14.269490 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:14.270709 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:14.291043 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:14.295354 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:14.296068 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:14.327714 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:14.334548 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:14.338128 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:14.340307 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:14.347398 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:14.350174 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:14.351132 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:14.399241 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:14.402124 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:14.407440 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:14.415129 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:14.429157 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:14.431558 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:14.457626 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:00:14.524888 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:14.526801 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:14.527060 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:14.537035 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:14.540654 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:14.555617 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:14.643985 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:14.647892 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:14.652069 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:14.658352 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:14.662636 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:14.665952 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:14.745258 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:14.761015 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:14.785256 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:14.799618 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:14.805438 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:14.809131 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:14.809625 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:14.813148 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:14.814749 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:00:14.817402 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:14.827596 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:14.833149 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:14.835003 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:00:14.840224 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:14.846844 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:14.864418 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:14.869267 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:14.883293 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:14.888984 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:14.900517 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:14.901268 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:14.909209 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:15.027374 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:15.030671 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.032442 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:15.033001 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:15.038565 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:15.056679 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:15.065649 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:15.068070 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:15.070058 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:00:15.075600 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:15.085263 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:15.095479 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.095674 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:15.187762 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:15.189378 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:00:15.218932 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:15.230584 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:15.234598 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:15.245376 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:00:15.246668 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:15.248245 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:15.260063 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:00:15.269609 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:15.270527 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:00:15.273394 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.282680 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:15.286073 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:15.286499 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.293179 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:15.298787 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:15.311381 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:15.320042 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:15.322923 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.326400 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:15.339762 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:15.348842 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:15.350145 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:00:15.366618 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:15.367077 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:15.389335 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:15.395617 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:15.420902 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:00:15.455132 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:15.460707 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:15.467705 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:00:15.472249 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:15.480963 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:15.507931 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:15.509212 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.519845 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:15.525902 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:15.526625 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:15.528057 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:15.528427 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:00:15.529440 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:00:15.533301 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.559864 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:15.568322 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 08:00:15.581367 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 85 similar messages]
W20240501 08:00:15.582046 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:15.591406 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 08:00:15.592280 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:15.593586 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 08:00:15.594367 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:15.624321 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:15.650671 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:15.651257 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:15.658612 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:15.658731 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:15.659145 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:15.674685 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:15.693152 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:00:15.703120 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:15.734042 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:15.738121 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:15.743016 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:15.743630 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.756474 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:15.756834 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:00:15.759354 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:15.762879 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:15.785858 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:15.804538 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:15.839218 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:15.840065 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:00:15.842286 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:15.845948 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:15.848418 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:15.866045 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.866140 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:15.874244 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:00:15.882751 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:00:15.915803 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:15.919189 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:15.920257 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:15.921088 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:15.928171 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:15.936712 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:16.012212 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:16.013095 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:00:16.014907 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:16.015532 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:16.022413 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.024384 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:16.025070 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:16.124333 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:16.129766 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:16.130976 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:16.138202 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:16.139663 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:16.142143 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:16.146947 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:16.151456 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:16.151777 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:16.158741 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 08:00:16.160527 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:00:16.181042 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:16.185743 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.190608 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:16.206180 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:16.297329 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:16.308933 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:16.311703 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:16.317278 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:16.325039 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:16.325892 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.326906 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:16.405440 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:16.416278 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:16.417414 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:16.418104 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:16.418377 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.419312 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:16.428012 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:00:16.486714 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:16.491508 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.493211 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:16.507970 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:16.519639 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:16.521915 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:16.592454 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:16.593515 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:16.599725 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:16.616271 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:16.616345 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:16.618326 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:16.622761 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.643049 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.650404 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:16.668243 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:16.670958 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:16.672680 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:16.690291 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:16.739068 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:16.743366 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:16.745191 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:16.746912 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:16.748503 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.761843 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:16.812896 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:16.831123 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:16.834822 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:16.835114 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:16.835281 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:16.853730 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:16.862066 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:16.926501 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:16.937624 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:16.938803 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:16.939441 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:16.946380 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:16.968927 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:16.982090 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:00:17.016211 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:17.016367 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:17.021009 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:17.025467 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:17.027495 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:17.038728 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:17.086488 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:17.091951 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:17.093281 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:17.106091 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:17.108884 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:17.110621 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:00:17.129256 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:17.183900 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:17.184070 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:17.187377 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:17.187642 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:17.188357 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:17.199892 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:17.205365 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:17.253567 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:17.256654 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:00:17.261039 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:17.262552 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:17.264847 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:17.273070 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:17.282618 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:17.294893 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:00:17.359020 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:17.366685 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:17.367712 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:17.372113 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:17.372169 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:17.372505 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:17.405177 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:17.459359 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:17.460059 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:17.461990 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:17.463232 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:17.463729 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:17.475494 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:17.542227 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:17.550941 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:17.552647 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:17.553151 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:17.567756 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:17.579607 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:17.592481 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:17.598526 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:17.602409 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:17.607429 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:17.611013 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:17.638734 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:17.646780 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:17.709514 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:17.712065 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:17.724254 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:17.726749 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:17.726786 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:17.734653 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:17.737341 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:17.738034 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:00:17.738679 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:17.792938 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:17.794252 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:17.795583 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:17.810354 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:17.821056 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:17.826928 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:17.842406 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:17.863996 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:17.893420 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:17.938170 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:17.938453 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:00:17.939707 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:17.948103 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:17.951135 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:17.952136 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:00:17.954955 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:17.958869 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:17.960400 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:17.961889 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:00:17.962268 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:17.971275 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:00:17.985517 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:17.986078 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:18.000888 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:18.001401 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:18.015131 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:18.066897 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:18.086067 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:18.116511 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:18.118477 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:18.127508 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:00:18.128625 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:18.135608 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:18.137986 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:18.143167 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:18.148521 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:18.150169 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:18.150907 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:18.155570 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:18.162395 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:18.177953 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:18.199934 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:18.207530 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:18.223156 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:18.229658 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:18.238155 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:18.239909 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:18.252255 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:18.257519 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:18.273836 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:00:18.312290 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:18.317588 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:18.340198 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:18.343238 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:00:18.345054 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:00:18.369145 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:18.380091 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:18.381108 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:18.384523 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:18.384855 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:18.399461 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:18.401517 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:18.408860 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:18.411635 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:18.412919 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:18.423856 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:18.426960 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:18.427747 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:18.441035 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:18.451305 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:18.454281 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:18.593884 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:18.598186 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:18.600066 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:18.609843 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:18.609876 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:18.613215 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:18.620911 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:18.630736 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:18.633394 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:18.644064 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:18.648800 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:18.659008 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:18.677836 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:18.783161 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:00:18.797791 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:18.802945 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:18.806491 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:18.820255 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:18.823684 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:18.831377 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:18.836022 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:00:18.840173 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:18.840654 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:18.847659 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:18.853757 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:18.867635 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:18.873306 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:18.887187 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:18.942059 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:18.942143 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:18.959340 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:18.961817 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:18.976635 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:18.989302 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:19.039382 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:19.042335 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:19.045323 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:19.047834 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:19.050010 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:19.067700 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:19.068111 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:19.071228 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:19.077754 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:19.089202 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:19.092916 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:19.109591 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:19.205073 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:19.206826 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:19.210863 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:19.214582 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:19.221433 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:19.221755 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:19.238656 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:19.323312 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:19.331004 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:19.332628 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:19.342092 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:19.350693 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:19.366497 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:00:19.378882 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:19.379555 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:19.380725 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:19.393137 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:19.395898 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:19.410221 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:19.437685 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:19.477681 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:19.479125 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:19.480024 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:19.484441 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:19.486593 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:19.500814 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:19.502791 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:19.600840 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:19.606527 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:19.607264 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:19.609498 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:19.621899 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:00:19.622983 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:19.638020 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:19.691982 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:19.711158 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:00:19.757727 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:19.759115 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:19.763259 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:19.767289 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:19.770479 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:19.770880 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:19.810974 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:19.815093 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:19.825877 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:19.827176 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:19.828853 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:19.843015 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:19.848410 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:19.849404 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:19.855284 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:19.871723 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:19.891829 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:00:19.896139 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:19.901235 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:19.934222 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:20.000530 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:20.010589 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:20.013685 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:20.014372 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:20.017462 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:20.020221 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:20.024509 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:20.025800 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:20.035058 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:20.042635 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:20.043617 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:20.079957 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:20.117914 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:20.193116 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:00:20.204138 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:20.207809 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:20.208504 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:20.209789 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:20.221484 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:20.226506 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:20.234100 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:20.237884 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:20.240624 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:20.244076 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:20.258239 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:20.276479 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:00:20.283104 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:20.291221 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:20.303951 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:00:20.306187 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:20.319553 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:00:20.321626 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:20.330758 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:20.334704 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:20.337172 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:20.341245 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:20.359784 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:20.372290 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:20.373849 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:20.416131 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:00:20.443300 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:20.444003 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:00:20.449122 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:20.468217 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:20.470572 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:20.470659 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:20.483884 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:20.492054 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:20.495270 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:20.500497 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:20.503681 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:00:20.523397 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:20.523488 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:20.535411 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:20.563477 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:20.592232 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:00:20.592926 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:20.601346 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:00:20.602285 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:20.611900 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:00:20.612821 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:20.613016 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:20.630368 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:00:20.637871 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:20.640043 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:20.643837 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:20.645309 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:20.650055 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:20.659866 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:00:20.661195 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:00:20.668359 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:20.704680 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 08:00:20.705111 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:20.707921 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:20.720780 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:20.727562 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:20.735595 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:00:20.736706 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:20.740006 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:20.760529 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:20.769254 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:20.778806 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:20.779321 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:20.779706 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:20.787724 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:20.794461 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:20.805980 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:20.812703 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:00:20.822063 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:20.864979 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:20.883119 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:00:20.902777 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:20.905997 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:20.909945 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:20.911391 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:20.925256 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:20.926538 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:20.937723 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:20.939098 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:20.951069 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:20.953966 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:20.959025 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:20.974187 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:21.042032 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:00:21.082366 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:21.083729 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:21.083801 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:21.084537 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:21.087498 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:21.089535 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:21.131204 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:21.137867 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:00:21.164135 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:21.165344 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:21.165843 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:21.167601 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:21.182304 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:21.182626 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:21.234961 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:00:21.279567 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:21.282455 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:21.284914 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:21.286473 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:21.297633 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:21.311653 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:21.363817 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:21.429040 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:21.436513 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:21.439239 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:21.439589 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:21.439994 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:21.445019 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:21.479003 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:21.488561 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:21.488915 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:21.498912 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 08:00:21.504771 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:21.509037 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:21.522413 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:21.613797 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:21.617118 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:21.624150 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:21.625403 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:21.626632 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:21.630875 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:21.632539 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:21.633826 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:21.657392 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:21.665940 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:21.666723 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:21.683013 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:21.718945 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:21.741024 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:21.748889 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:21.751711 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:21.759037 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:21.759358 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:21.761322 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:21.848924 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:21.855463 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:21.857491 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:21.862838 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:21.867239 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:21.868405 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:21.895545 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:21.916518 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:21.918596 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:21.923314 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:21.930498 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:21.930929 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:21.931921 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:21.968832 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:00:22.010915 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:22.046777 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:00:22.076236 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:22.088181 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:22.089403 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:22.096705 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:22.100608 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:22.131481 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:22.134775 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:22.138543 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:22.141991 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:22.144316 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:22.151268 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:22.161240 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:22.169224 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:22.186748 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:22.191957 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:22.195761 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:22.200528 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:22.221933 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:22.306990 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:00:22.331995 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:22.334487 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:22.337363 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:22.351781 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:22.359190 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:22.360148 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:22.382053 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:22.384189 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:22.384482 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:22.391139 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:22.397272 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:22.416067 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:22.446846 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:22.484637 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:00:22.520123 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:22.527081 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:22.538970 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:22.540017 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:22.566560 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:22.577281 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:22.578295 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:22.588078 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:22.590675 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:22.608561 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:22.612051 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:22.614809 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:22.620024 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:22.628150 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:22.628278 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:22.635344 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:22.635999 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:22.645074 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:00:22.661934 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:22.663626 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:22.728588 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:22.752665 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:00:22.784466 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:22.785454 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:22.790131 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:22.797147 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:00:22.798527 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:22.805830 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:22.809000 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:22.820544 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:22.826800 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:22.832438 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:22.842351 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:22.856451 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:22.858901 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:22.862162 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:22.862710 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:22.871083 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:22.884034 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:22.942196 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:22.961422 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:00:22.977193 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:22.979074 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:22.981194 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:22.991443 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:22.997515 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:22.999671 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:23.041357 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:23.083066 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:23.134084 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:23.155107 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:23.171399 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:00:23.183898 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:23.188921 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:23.192552 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:23.194263 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:23.194685 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:23.199465 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:23.213323 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:23.214355 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:23.223503 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:00:23.226223 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:23.227515 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:23.237866 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:23.240366 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:23.249701 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:00:23.256712 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:23.271780 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:23.295046 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:23.353137 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:23.381668 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:23.386176 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:23.387931 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:00:23.389257 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:23.409776 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:23.411609 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:23.411767 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:23.430657 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:23.435405 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:23.449882 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:23.465607 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:23.465916 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:23.465997 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:23.470774 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:23.473976 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:23.482237 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:23.482254 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:00:23.490964 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:00:23.491741 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:23.492342 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:23.500416 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:23.508752 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:23.521822 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:00:23.639856 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:23.644377 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:23.648458 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:23.649641 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:23.657109 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:23.668332 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:23.678632 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:23.680404 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:23.681037 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:23.693157 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:00:23.694897 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:23.728576 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:23.729285 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:23.798682 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:23.822232 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:23.825387 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:23.836093 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:00:23.836786 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:23.857372 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:23.857869 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:23.865080 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:23.868425 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:23.870443 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:23.870805 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:23.872291 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:23.888125 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:23.905489 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:23.911893 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:23.914577 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:23.928809 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:23.931082 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:23.956099 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:23.960644 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:23.961742 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:24.064879 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:24.065354 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:24.065568 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:24.070509 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:24.076591 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:24.078150 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:24.136888 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:24.140584 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:24.140966 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:24.141606 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:24.145906 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:24.153192 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:24.229734 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:24.253826 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:24.256400 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:24.256541 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:24.259416 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:24.277439 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:24.279635 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:24.282255 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:24.303264 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:00:24.341087 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:24.345768 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:00:24.350034 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:24.351248 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:24.361428 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:24.361657 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:24.388326 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:24.389863 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:24.393702 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:24.408339 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:24.416208 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:24.425583 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:24.426139 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:24.523204 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:24.528889 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:24.532717 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:24.537195 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:24.538636 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:24.566471 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:24.573904 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:24.577513 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:24.589898 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:24.591634 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:24.595685 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:24.597955 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:24.710911 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:24.719170 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:24.721154 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:24.721859 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:24.724684 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:24.748436 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:24.762735 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:24.772786 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:24.777202 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:24.783421 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:24.792580 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:24.792701 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:24.849923 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:24.867475 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:24.874208 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:24.889497 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:24.892175 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:24.900480 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:24.904454 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:24.918365 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:24.919349 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:25.003297 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:25.010574 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:25.011379 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:25.012912 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:25.021227 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:25.025059 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:25.067461 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:25.068267 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:25.123585 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:00:25.127935 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:25.132128 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:25.138525 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:25.145757 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:25.153548 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:25.171239 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:25.187041 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:00:25.194995 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:25.200986 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:25.210223 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:25.214836 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:25.221375 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:25.229614 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:25.230212 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:25.238780 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:25.239177 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:25.239178 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:25.251546 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:25.258314 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:25.267920 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:00:25.270783 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:25.302357 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:25.315807 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:25.317606 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:25.343429 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:25.376706 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:25.376727 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:25.378402 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:25.391991 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:25.393594 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:00:25.395731 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:25.397840 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:25.442245 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:25.448390 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:25.451457 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:25.452140 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:25.462674 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:25.466897 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:00:25.489761 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:25.493494 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:25.558351 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:00:25.560668 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:25.563031 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:25.566941 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:25.575244 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:25.576869 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:25.580350 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:00:25.606880 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:25.610638 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:00:25.619482 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:00:25.640031 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:25.640167 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:25.655050 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:25.658695 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:25.671000 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:25.695484 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:25.703150 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:00:25.703743 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:00:25.706609 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:25.723676 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:25.726548 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:25.727139 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:25.728881 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:25.744051 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:25.768811 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:00:25.769546 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:25.771486 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 08:00:25.799111 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:00:25.810499 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:25.812114 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:25.819774 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:25.820704 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:25.824786 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:00:25.825562 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:25.827131 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:25.836987 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:25.886360 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:25.898363 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:25.898481 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:00:25.900336 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:25.910565 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:25.913990 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:25.922243 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 08:00:25.923070 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:25.923444 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:25.935364 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:25.998122 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:00:25.999219 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:26.004330 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:26.006503 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:26.010509 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:26.015986 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:00:26.028018 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:26.029976 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:26.040555 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:00:26.044308 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:00:26.082809 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:26.083243 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:26.092211 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:00:26.100847 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:26.115262 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:26.119537 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:26.139200 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:26.164072 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:26.168782 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:26.171149 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:26.171679 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:26.189038 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:26.198262 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:26.238859 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:26.239033 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:26.245704 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:26.247205 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:26.253919 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:26.267107 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:26.368700 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:26.369439 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:26.370721 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:26.371400 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:26.376346 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:26.392473 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:26.430486 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:26.460189 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:26.481683 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:26.481925 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:26.493134 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:26.495368 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:26.511924 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:26.529978 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:26.554621 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:00:26.580957 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:26.589610 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:26.591856 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:26.597693 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:26.599928 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:26.613831 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:26.615913 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:26.617457 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:26.635284 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:26.638703 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:26.644371 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:26.647271 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:26.731180 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:26.733899 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:26.737133 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:26.738593 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:26.742954 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:26.743587 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:26.836834 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:26.863747 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:26.872751 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:26.872818 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 08:00:26.873492 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:00:26.873767 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:26.883145 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:26.894343 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:26.896358 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:26.896701 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:26.897261 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:26.898814 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:26.915023 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:26.922618 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:26.937672 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:26.939227 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:26.982010 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:27.046859 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:27.052884 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:27.053743 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:27.060505 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:27.066660 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:27.067024 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:27.091836 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:27.094254 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:27.097683 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:27.125560 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:27.128336 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:27.138363 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:27.248759 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:27.256949 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:27.264008 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:27.264559 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:27.270288 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:27.292165 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:27.297577 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:27.305972 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:27.310298 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:27.318313 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:27.320130 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:27.335382 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:27.351158 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:27.361701 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:27.393959 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:27.402678 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:27.410724 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:27.413470 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:27.414101 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:27.414558 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:27.421787 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:27.509618 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:00:27.544138 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:27.546485 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:27.548583 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:27.548702 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:27.549202 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:27.570101 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:27.570868 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:27.577684 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:27.579676 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:27.594552 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:00:27.594848 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:27.608058 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:27.617374 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:00:27.618337 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:27.693709 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:00:27.718993 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:27.727846 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:27.736807 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:27.750602 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:27.751348 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:00:27.754328 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:27.774669 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:00:27.777158 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:27.795485 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:27.805819 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:27.807778 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:27.807962 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:27.812640 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:27.812810 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:27.814267 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:27.848999 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:00:27.861953 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:27.864995 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:00:27.870105 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:27.876720 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:27.879694 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:27.890949 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:27.896775 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:27.949165 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:27.966172 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:27.973349 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:27.975812 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:00:27.992303 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:27.994323 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:27.995354 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:27.998373 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:27.999143 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:28.003387 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:28.006058 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:28.015436 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:28.117669 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:00:28.121999 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:28.122833 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:28.125771 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:28.128990 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:28.129186 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:28.138819 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:28.165302 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:28.193843 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:28.223896 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:28.225755 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:28.256625 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:28.256593 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:28.258215 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:28.262018 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:28.269181 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:00:28.269688 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:28.284679 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:28.307128 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:00:28.335536 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:28.336434 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:28.341356 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:28.344211 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:28.350693 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:28.355597 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:28.360816 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:28.369737 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:28.384299 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:28.384701 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:28.392020 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:00:28.394768 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:28.398566 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:28.436445 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:28.497912 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:28.506299 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:28.509498 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:28.513703 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:28.515806 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:28.519037 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:28.532785 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:28.549882 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:28.566884 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:00:28.585467 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:28.589653 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:28.593400 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:28.593713 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:28.595777 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:28.596735 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:28.631597 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:28.647679 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:28.706539 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:28.707420 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:28.708318 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:28.712653 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:28.713380 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:28.725204 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:28.772661 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:28.774449 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:28.778945 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:28.789125 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:28.789638 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:28.791111 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:28.803805 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:00:28.856587 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:00:28.928262 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:28.928608 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:28.930275 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:28.938917 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:28.941547 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:28.958496 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:28.967628 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:28.979853 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:28.981535 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:28.986054 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:28.988757 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:28.996042 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:28.999155 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:29.140144 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:29.146387 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:29.154587 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:29.157459 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:29.157584 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:29.165287 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:29.166049 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:29.179127 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:29.182797 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:29.198405 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:29.198647 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:29.200167 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:29.203033 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:29.224718 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:29.238173 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:00:29.240615 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:29.247958 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:29.248801 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:29.254123 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:29.257786 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:29.362326 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:29.368755 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:29.369179 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:29.372627 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:29.397173 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:00:29.404085 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:29.409607 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:29.420002 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:29.428095 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:29.428738 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:29.432734 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:29.435606 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:29.442184 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:29.449810 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:29.564713 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:29.570801 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:29.584235 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:29.589341 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:29.597489 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:29.600162 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:29.604136 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:29.613492 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:29.619144 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:29.619670 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:29.637943 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:29.655441 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:29.658641 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:29.662581 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:29.678603 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:29.694279 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:29.701057 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:29.704576 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:29.821287 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:29.822060 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:29.824122 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:29.831113 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:29.832749 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:29.840788 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:29.846606 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:29.880383 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:29.938649 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:29.941321 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:29.943404 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:29.953090 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:29.956391 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:29.957127 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:29.971065 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:29.971802 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:29.996049 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:30.001314 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:00:30.081013 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:30.089915 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:30.097770 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:30.113834 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:30.119609 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:30.127921 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:30.133838 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:00:30.146564 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:30.146915 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:30.149574 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:30.158108 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:30.158819 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:30.162194 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:30.167264 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:30.170723 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:00:30.176981 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:30.180004 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:30.187052 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:00:30.191365 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:30.191919 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:00:30.194294 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:30.195437 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:30.283057 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:30.326135 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:30.331933 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:30.343902 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:30.346674 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:30.350044 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:30.365461 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:30.370285 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:30.377616 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:30.384065 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:30.384649 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:30.397277 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:30.399034 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:30.404727 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:30.417722 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:30.425436 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:30.428925 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:30.455303 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:30.484771 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:30.495508 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:00:30.517097 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:30.523763 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:30.527705 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:00:30.528484 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:30.529454 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:30.530436 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:30.546527 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:30.555251 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:00:30.584309 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:30.584426 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:30.589854 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:30.600215 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:00:30.600749 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:30.601025 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:30.605399 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:30.616062 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240501 08:00:30.617018 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:30.619226 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:30.704916 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:00:30.733328 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:30.737326 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:30.741225 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:30.745420 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:30.749507 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:30.754602 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:30.756892 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 08:00:30.766741 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:30.778192 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:00:30.779421 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:30.780299 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:30.787441 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:30.798509 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:00:30.800143 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:30.827761 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:30.833859 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:30.885540 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:30.886281 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:30.888896 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:30.896365 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:30.901196 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:00:30.901760 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:30.922906 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:30.947890 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:00:30.953652 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:30.988430 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:30.991137 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:30.992036 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:30.993650 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:31.008569 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:31.012434 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:31.033239 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:31.048508 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:31.050158 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:00:31.051290 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:31.054662 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:31.060261 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:31.060742 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:31.173009 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:31.173468 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:31.177060 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:31.181883 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:31.184491 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:00:31.193100 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:31.213868 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:31.214139 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:31.226341 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:31.227301 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:31.236632 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:31.244073 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:31.244488 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:00:31.251648 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:31.271909 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:31.288589 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:31.291828 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:31.293032 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:31.293465 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:31.305385 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:31.305528 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:00:31.318068 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:31.404809 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:00:31.431192 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:31.432329 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:31.447649 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:31.449380 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:31.472925 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:31.473876 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:31.476572 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:31.478215 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:31.489589 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:31.500079 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:31.503716 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:31.522629 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:31.532191 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:31.558809 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:00:31.616528 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:31.627859 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:31.628620 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:31.628790 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:31.631876 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:31.640401 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:00:31.654470 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:31.693699 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:31.698194 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:31.710345 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:31.711618 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:31.713385 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:31.713383 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:31.768298 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:00:31.801213 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:31.805297 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:31.807070 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:31.820205 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:31.828779 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:31.829499 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:31.831804 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:31.883955 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:31.884025 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:31.884887 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:31.887897 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:31.891132 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:31.897750 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:31.974848 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:31.979166 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:31.979558 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:31.988106 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:31.988749 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:31.998317 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:32.007483 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:32.072062 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:32.075611 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:32.080278 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:32.080699 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:32.084512 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:32.111109 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:32.162513 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:32.164528 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:32.171195 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:32.172202 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:32.173357 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:32.176602 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:32.222324 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:00:32.237890 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:32.240731 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:32.247505 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:32.248034 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:32.248203 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:32.257364 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:32.327287 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:32.329006 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:32.331225 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:32.332512 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:32.341307 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:32.356389 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:32.364168 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:32.372257 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:32.410187 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:32.418903 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:32.421185 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:00:32.455588 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:32.455789 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:32.456462 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:32.470949 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:32.476387 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:32.483762 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:32.538125 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:32.555261 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:32.561229 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:32.564533 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:00:32.572445 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:32.573560 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:32.576131 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:32.583128 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:32.617156 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:32.639400 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:32.641216 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:32.647970 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:32.658975 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:32.659325 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:32.672077 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:32.696981 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:00:32.724386 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:32.731331 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:32.731876 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:32.732939 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:32.734328 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:32.748941 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:32.767460 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:00:32.859067 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:32.859504 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:32.867599 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:32.870234 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:32.871265 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:32.873973 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:32.874569 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:32.875485 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:32.889099 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:32.894876 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:32.899538 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:32.899879 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:32.912950 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:32.918936 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:32.960618 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:32.983994 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:33.003943 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:33.042376 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:33.043309 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:33.045264 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:33.062288 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:33.071102 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:00:33.071514 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:33.078366 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:33.086220 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:33.096027 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:33.096642 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:00:33.097230 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:33.103694 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:33.106330 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:00:33.108613 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:33.111272 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:33.115531 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:00:33.146662 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:33.148023 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:33.151954 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:33.156672 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:33.167318 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:33.176149 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:00:33.179422 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:33.180853 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:00:33.202761 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:33.252372 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:00:33.285089 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:33.290585 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:33.293249 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:00:33.293784 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:33.293861 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:33.301156 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:33.306254 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:33.388850 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:33.423182 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:33.423826 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:33.434844 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:33.440536 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:33.452183 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:33.463824 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:33.464092 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:00:33.466992 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:33.509836 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:00:33.512545 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:33.517423 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:33.532778 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:33.533535 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:33.540647 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:33.563756 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:33.573644 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:33.582479 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:33.584420 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:33.585906 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:33.586511 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:33.599351 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:33.603607 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:33.682909 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:33.689276 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:33.702765 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:33.702838 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:33.704901 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:33.725633 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:00:33.728893 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:33.738924 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:00:33.773576 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:33.775442 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:33.783977 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:33.785852 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:33.786195 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:33.809648 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:33.896032 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:33.898038 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:33.900425 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:33.905249 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:33.914743 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:33.919023 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:33.920805 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:33.970227 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:33.974294 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:33.978770 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:33.991852 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:34.001567 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:34.002167 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:34.041474 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:34.042694 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:34.043752 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:34.052174 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:34.054920 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:34.063830 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:34.151681 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:34.152681 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:34.158620 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:34.160629 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:00:34.163964 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:34.167249 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:34.169118 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:34.181478 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:34.213626 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:34.216531 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:34.217335 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:34.217825 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:34.217908 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:34.228292 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:34.253993 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:34.342753 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:00:34.350225 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:00:34.356688 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:34.356859 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:34.357934 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:34.368050 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:34.368901 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:34.373416 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:34.385598 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:34.424369 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:34.424610 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:34.432793 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:34.439385 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:34.444386 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:34.444689 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:34.529196 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:34.546746 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:34.549625 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:34.549671 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:34.551890 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:34.570881 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:34.588277 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:34.589744 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:34.595965 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:34.601691 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:34.603654 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:34.623251 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:34.678151 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:34.732837 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:34.738932 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:34.743148 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:34.743567 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:34.749341 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:34.772037 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:34.779151 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:34.785507 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:34.785794 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:34.785980 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:34.787112 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:34.814446 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:34.814711 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:34.885149 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:34.894562 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:34.895752 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:34.898370 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:34.902297 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:34.912681 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:34.969345 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:34.972146 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:34.972892 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:34.973884 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:34.981817 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:34.982247 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:34.993042 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:35.013423 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:35.016073 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:00:35.089497 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:35.093950 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:35.099762 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:35.103478 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:35.107146 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:35.125171 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:35.139660 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:35.140178 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:00:35.174603 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:35.177898 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:35.178334 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:35.179611 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:35.181702 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:35.192742 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:35.215759 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:35.226365 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:00:35.287312 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:35.291026 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:35.305270 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:00:35.315620 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:35.331943 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:35.334832 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:35.337644 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:35.339154 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:35.339973 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:35.363978 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:35.367823 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:35.377854 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:35.381479 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:35.413697 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:35.425149 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:35.427947 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:35.443343 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:35.443924 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:35.446345 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:35.448566 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:35.476290 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:35.484081 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:35.505710 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:35.530696 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:35.534102 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:35.535882 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:00:35.561548 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:35.568286 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:35.574529 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:35.575044 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:35.577275 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:35.597812 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:00:35.603185 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:35.609376 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:35.609892 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:35.629137 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:35.635093 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:35.636891 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:35.653447 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:35.654397 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:35.687775 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240501 08:00:35.708209 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:35.741782 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:35.775735 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:35.776552 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:00:35.778534 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:35.781095 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:35.785928 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:35.786063 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:35.788097 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:35.789155 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:00:35.807088 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:35.816299 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:35.820247 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:35.826751 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:35.831602 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:35.845329 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:35.855370 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 08:00:35.879400 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:35.882036 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:35.887442 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:35.891172 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:35.907258 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:35.907646 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240501 08:00:35.909530 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:35.909922 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:00:35.913089 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:35.917950 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:35.948280 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:35.961655 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:35.992864 11598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240501 08:00:35.993640 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:00:36.011916 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:36.016993 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:36.031409 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:36.044788 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:36.051236 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:36.059254 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:36.059816 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:36.063416 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:36.072816 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:36.074594 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:36.077117 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:36.083201 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:36.091609 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:00:36.093592 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:36.171823 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:00:36.197603 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:36.202378 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:36.202836 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:36.203904 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:36.207039 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:36.207453 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:36.237519 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:36.267872 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:36.271744 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:36.275808 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:36.278492 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:36.280882 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:36.295254 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:36.304807 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:36.397732 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:36.398347 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:36.402570 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:36.404191 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:36.406163 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:36.421594 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:36.473246 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:36.474570 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:36.474994 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:36.476382 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:36.487207 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:36.505331 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:36.550546 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:36.612600 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:36.617096 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:36.617123 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:36.632867 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:36.634845 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:36.650619 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:36.663231 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:36.668553 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:36.680660 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:36.688942 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:36.708568 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:36.713946 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:36.799495 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:36.800874 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:36.805927 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:36.806231 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:36.810060 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:36.814744 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:36.817473 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:36.827335 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:36.855832 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:36.857637 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:36.868150 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:36.872231 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:36.874912 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:36.877065 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:36.895737 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:00:36.929051 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:00:37.003120 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:37.004037 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:37.006040 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:37.010650 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:37.012917 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:00:37.014185 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:37.044781 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:37.052523 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:37.059540 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:37.059830 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:37.059913 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:37.061029 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:37.081607 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:37.123925 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:37.136171 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:37.138306 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:37.140102 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:37.150815 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:37.163975 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:37.252640 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:00:37.253778 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:37.254110 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:37.263161 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:37.264336 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:37.265038 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:37.265039 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:37.284315 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:37.286304 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:37.295435 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:37.296267 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:37.306834 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:37.307883 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:37.321084 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:37.398542 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:37.455495 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:37.455924 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:37.456992 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:37.463438 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:37.464020 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:37.470616 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:37.482618 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:37.498164 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:37.500119 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:37.500891 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:37.508486 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:37.539673 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:37.540261 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:37.558910 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:37.572856 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:37.580583 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:37.580586 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:37.582005 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:37.587706 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:37.589596 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:37.604050 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:37.608219 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:37.662652 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:37.723682 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:37.725137 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:37.730870 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:37.735363 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:37.739450 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:37.745033 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:37.745504 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:37.747193 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:37.752846 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:37.754858 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:37.772287 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:37.776227 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:37.779526 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:37.794574 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:37.799902 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:37.836987 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:37.840478 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:37.898830 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:37.903133 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:37.912835 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:37.916153 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:37.922632 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:37.935004 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:37.942667 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:37.946683 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:37.949026 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:37.949429 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:37.952658 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:37.972898 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:37.979317 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:38.003077 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:38.003794 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:38.010759 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:38.017554 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:38.019980 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:38.021183 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:38.021811 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:38.050688 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:38.065928 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:38.088482 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:38.097865 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:38.145038 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:38.146507 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:38.147048 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:38.147593 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:38.147955 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:38.163925 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:38.175257 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:38.197885 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:38.221403 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:38.303908 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:38.305375 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:38.305678 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:38.311753 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:38.347226 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:38.368348 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:38.376981 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:38.377142 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:38.377645 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:38.381927 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:38.383003 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:38.429939 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:38.431591 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:38.432248 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:38.438606 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:38.444031 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:38.458516 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:38.525427 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:38.529840 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:38.534619 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:38.536392 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:38.538467 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:38.538707 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:38.546509 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:38.546903 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:38.561897 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:38.614909 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:38.615154 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:38.615721 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:38.617277 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:38.626914 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:38.647964 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:38.670887 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:00:38.711858 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:38.713989 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:38.717417 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:38.717870 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:38.720494 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:38.730494 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:38.743372 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:00:38.751870 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:38.785566 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:38.816697 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:38.819497 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:38.824321 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:38.849823 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:38.854965 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:38.855712 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:38.862118 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:38.916601 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:38.920418 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:38.926056 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:38.926916 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:38.937345 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:38.951930 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:38.988642 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:38.991057 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:38.996387 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:38.998044 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:38.999691 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:39.045114 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:39.052444 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:39.056070 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:39.061271 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:39.075042 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:39.084707 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:39.093590 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:39.147787 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:39.166803 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:39.173887 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:39.177322 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:39.179021 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:39.194190 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:39.199822 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:39.224431 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:39.249671 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:39.259140 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:39.261739 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:39.262682 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:39.263576 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:39.268723 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:39.308743 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:39.308892 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:39.358220 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:39.359299 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:00:39.360577 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:39.362710 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:39.364873 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:39.374282 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:39.382380 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:39.450793 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:39.454486 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:39.464090 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:39.466185 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:39.466312 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:39.467831 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:39.480219 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:39.498135 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:39.503598 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:39.503923 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:39.513353 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:39.513944 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:39.621942 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:39.626653 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:39.631098 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:39.642977 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:39.645036 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:39.662402 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:39.679971 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:39.704622 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:39.710173 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:39.710889 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:39.725098 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:39.726629 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:39.744405 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:39.768656 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:39.801873 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:39.808156 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:39.813944 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:39.818934 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:39.835465 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:39.839013 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:39.843521 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:39.876520 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:39.880383 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:00:39.884497 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:39.886984 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:39.891650 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:39.898532 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:39.914124 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:39.945387 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:39.953018 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:39.953040 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:39.953213 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:39.954507 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:39.955523 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:40.099277 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:40.104348 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:40.137356 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:40.138669 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:00:40.151364 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:40.154989 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:40.157900 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:40.160797 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:40.160797 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:40.179107 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:40.204960 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:40.222780 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:40.229295 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:40.282333 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:40.284188 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:40.285319 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:40.302006 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:40.306609 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:40.313155 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:40.322912 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:40.332072 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:40.340490 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:40.341033 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:40.363519 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:40.364629 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:40.375923 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:40.424340 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:40.444319 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:40.455250 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:00:40.485085 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:40.487174 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:40.494633 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:40.494966 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:40.501387 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:40.505564 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:40.509408 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:40.519269 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:40.523741 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:40.530872 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:00:40.536734 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:40.547066 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:40.559254 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:40.577549 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:40.600185 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:00:40.605360 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:40.612462 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:40.613795 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:40.615554 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:40.618014 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:40.625088 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:40.632480 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:40.639863 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:40.698846 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:00:40.699641 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:40.702826 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:40.703966 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:00:40.746898 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:40.747197 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:40.751080 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:40.751471 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:40.756057 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:40.763017 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:40.765717 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:40.780833 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:40.784219 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:40.797912 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:40.799227 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:00:40.810520 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:40.818392 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:40.853394 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:40.876686 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:00:40.877461 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:40.890836 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:00:40.927834 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:40.931943 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:40.934278 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:40.935071 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:40.938823 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:40.946063 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:40.991382 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:40.996133 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:41.002619 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:41.014051 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:41.015554 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:41.023093 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:41.028976 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:41.035606 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:41.042362 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:41.047159 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:41.057907 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:41.081881 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:41.085561 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 08:00:41.092157 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:41.098063 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:00:41.123504 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:41.130223 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 08:00:41.159906 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:00:41.176489 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:41.177983 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:41.184665 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:41.186594 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:41.194243 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:41.196645 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:41.207069 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:00:41.207460 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:41.249771 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:41.251525 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:41.254626 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:41.254810 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:41.260809 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:41.267690 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:41.286451 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:00:41.376056 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:41.393693 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:41.396708 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:41.397717 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:41.398175 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:41.408900 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:41.455149 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:41.463061 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:41.465150 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:41.468361 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:41.477238 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:41.478724 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:41.497627 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:41.530138 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:41.530411 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:41.540169 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:41.542222 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:00:41.545264 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:41.545461 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:41.556391 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:41.623430 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:41.630978 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:41.638175 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:41.640782 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:41.641783 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:41.643312 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:41.644676 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:41.726819 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:41.728011 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:41.729559 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:41.732618 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:41.739773 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:41.748862 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:41.806730 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:00:41.867883 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:00:41.872396 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:41.873832 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:41.881650 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:41.882696 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:41.883323 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:41.887842 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:41.889286 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:41.899962 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:41.906533 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:41.908878 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:41.910614 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:41.928246 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:41.933316 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:42.032163 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:42.034798 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:42.035454 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:42.036211 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:42.037369 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:42.054646 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:42.075584 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:42.096597 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:00:42.152288 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:42.153569 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:42.154162 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:42.157748 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:42.157851 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:42.179311 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:42.279592 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:42.284577 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:00:42.286857 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:42.293543 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:42.304687 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:42.312753 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:42.325549 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:42.332413 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:42.336776 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:42.338729 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:42.343406 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:42.376374 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:42.391939 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:42.396456 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:42.400710 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:42.405449 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:42.412477 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:00:42.415419 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:42.417863 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:42.433429 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:42.460150 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:42.470268 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:42.481096 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:00:42.526002 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:42.530009 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:42.530145 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:42.530807 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:42.538590 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:42.544385 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:42.574970 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:42.586701 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:00:42.591054 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:42.590957 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:42.599530 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:42.599843 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:42.602667 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:00:42.614755 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:42.617712 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:42.640009 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:42.640545 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:00:42.762763 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:42.764808 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:42.769425 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:42.771757 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:42.775061 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:42.776073 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:42.777681 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:42.783444 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:42.786847 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:42.787109 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:42.801702 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:42.810596 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:42.825299 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:42.831997 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:42.839157 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:42.842566 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:00:42.846621 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:42.879081 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:42.901377 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:42.990510 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:42.993026 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:42.996852 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:42.999660 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:43.012367 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:43.013092 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:43.017714 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:43.019237 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:43.030022 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:43.036648 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:43.038245 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:43.039958 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:43.043857 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:00:43.044052 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:43.046137 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:00:43.075331 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:43.099310 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:43.120258 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:43.134313 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:43.144707 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:43.148236 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:00:43.196055 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:43.213069 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:43.224184 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:43.224907 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:43.231117 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:43.234874 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:43.242173 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:43.251667 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:43.254384 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:43.258904 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:43.267055 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:43.285975 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:43.287818 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:43.290503 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:43.320189 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:00:43.330093 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:43.345039 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:43.349784 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:43.365844 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:43.379544 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:43.394608 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:00:43.421969 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:43.422621 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:43.425632 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:43.433638 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:43.441371 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:43.442081 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:43.451434 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:43.507268 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:43.538228 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:43.541026 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:43.549261 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:43.549948 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:43.550968 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:43.555676 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:43.560079 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:43.564081 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:43.601989 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:43.605453 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:43.611465 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:43.645664 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:00:43.676023 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:43.683950 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:43.708940 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:43.719059 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:43.730298 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:43.734246 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:43.744639 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:43.748221 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:43.771518 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:43.773969 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:43.776513 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:43.779201 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:43.781442 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:43.800830 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:43.803457 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:43.807819 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:43.836558 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:43.845615 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:43.919001 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:43.921036 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:43.922050 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:43.923022 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:43.937398 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:43.946604 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:43.963352 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:43.973543 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:43.994024 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:00:44.040285 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:44.041026 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:44.041078 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:44.046823 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:44.048419 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:44.050501 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:00:44.067427 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:44.086372 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:00:44.138149 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:44.190637 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:44.194705 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:44.196204 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:44.203114 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:44.206357 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:44.212316 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:44.218407 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:44.222988 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:44.237613 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:44.240165 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:44.241595 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:44.252733 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:44.253911 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:44.270736 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:44.278318 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:44.299063 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:44.303294 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:44.305704 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:44.419275 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:44.419823 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:44.423684 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:44.430315 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:44.439987 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:44.441989 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:44.477933 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:44.483939 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:44.485392 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:44.485589 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:44.491037 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:44.510607 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:44.518052 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:44.595172 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:44.597939 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:44.598205 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:44.598930 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:44.601459 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:44.623148 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:44.631253 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:44.638751 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:00:44.673213 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:44.674079 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:44.674651 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:44.689994 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:44.704856 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:44.706152 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:44.710780 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:44.715219 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:44.723495 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:44.726828 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:44.735165 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:44.745400 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:44.850160 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:44.852921 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:44.859706 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:44.863935 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:44.870631 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:44.880712 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:44.883226 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:44.894059 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:00:44.915042 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:44.919904 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:44.927094 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:44.927563 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:44.930611 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:44.944012 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:44.963491 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:44.967478 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:44.981575 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:45.021157 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:00:45.033401 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:45.034353 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:45.034683 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:45.036325 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:45.073088 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:45.078310 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:45.078516 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:45.080564 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:45.089107 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:45.097558 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:45.101128 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:45.102814 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:45.123498 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:45.124341 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:45.213109 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:00:45.226384 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:45.228489 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:45.230317 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:45.234923 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:45.237397 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:45.241765 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:45.250074 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:00:45.297020 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:45.308679 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:45.314270 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:45.317209 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:45.323720 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:45.324949 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:45.410723 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:00:45.412314 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:45.419567 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:45.426595 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:45.426885 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:45.428418 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:45.428762 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:45.522817 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:45.525633 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:45.528097 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:45.540589 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:45.546557 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:45.547385 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:45.548475 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:00:45.558615 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:45.561069 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:45.563836 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:45.564556 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:45.571709 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:45.592350 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:45.593252 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:45.654253 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:45.654807 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:45.661192 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:45.664701 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:45.672392 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:45.672770 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:45.678633 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:45.684449 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:45.686540 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:45.695031 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:45.704399 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:00:45.705086 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:45.719281 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:00:45.758342 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:45.759403 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:45.762650 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:45.765084 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:45.770902 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:45.790076 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:45.856732 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:00:45.891947 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:45.893162 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:45.895318 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:45.898185 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:00:45.899583 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:45.913712 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240501 08:00:45.915793 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:45.923537 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:45.947352 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:00:45.966954 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:45.967640 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:45.973949 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:45.974571 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:45.977089 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:45.986774 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:45.987958 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:46.009930 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:00:46.070516 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:46.072950 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:46.074033 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:46.079380 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:46.083637 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:46.095733 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:00:46.096071 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:46.108228 11598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:00:46.109019 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:46.118156 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:00:46.137982 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:46.140277 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:46.143260 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:46.146334 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:46.147099 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:46.148842 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:46.154559 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:46.171962 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:46.188055 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 08:00:46.188858 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:00:46.281630 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:46.289386 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:46.297065 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:46.302500 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:46.303673 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:00:46.313500 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:46.330768 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:46.331614 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:46.333024 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:46.340382 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:46.343099 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:46.351748 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:46.359311 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:46.433634 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:46.440013 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:46.460994 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:46.462131 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:46.465961 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:46.478677 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:46.523772 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:46.529354 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:00:46.552822 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:46.555292 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:46.556183 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:46.559278 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:46.569068 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:46.577627 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:46.580222 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:46.592157 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:46.594987 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:46.604722 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:46.606477 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:46.610057 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:46.645201 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:00:46.657017 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:46.678047 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:46.723294 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:46.723394 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:46.726370 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:46.726696 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:46.729668 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:46.742805 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:46.829844 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:46.832883 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:46.836426 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:46.839452 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:46.844871 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:46.855201 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:46.905205 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:46.906903 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:46.908366 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:46.913244 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:46.920637 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:46.923233 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:46.950917 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:00:46.997570 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:47.004671 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:47.016887 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:47.019666 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:47.025677 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:47.039064 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:47.067322 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:47.068377 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:47.072918 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:47.076994 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:47.079871 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:47.083032 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:47.098888 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:47.135385 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:47.205830 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:47.207178 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:47.221592 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:47.238559 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:47.241039 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:47.252341 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:47.255239 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:47.270823 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:47.279528 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:47.285486 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:47.286814 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:47.291632 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:47.300074 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:47.317757 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:47.318708 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:47.338972 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:47.347398 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:47.349913 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:47.372490 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:47.411316 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:47.477614 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:47.487784 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:47.488065 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:47.490166 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:47.491042 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:47.500749 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:47.514508 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:47.516811 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:47.518792 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:47.525446 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:47.529058 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:47.546548 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:47.546758 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:47.548105 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:00:47.569391 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:47.569664 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:47.570279 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:47.572108 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:47.614193 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:47.645691 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:47.679379 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:47.679641 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:47.682826 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:47.684821 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:47.696282 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:47.698096 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:47.698208 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:47.715193 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:47.716815 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:47.725885 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:47.734597 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:47.763456 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:47.764818 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:47.771148 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:47.883239 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:47.885792 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:47.887220 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:47.892684 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:47.908124 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:47.908149 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:47.908933 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:47.923266 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:47.931759 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:47.940848 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:47.947168 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:47.954484 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:47.967301 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:48.069466 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:48.072830 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:48.074546 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:48.078958 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:48.081117 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:48.089743 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:48.093228 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:48.104954 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:48.123275 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:48.125862 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:48.140970 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:48.144937 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:48.147646 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:48.151429 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:48.159659 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:48.161841 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:48.175987 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:48.176100 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:48.179771 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:48.188673 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:48.194747 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:48.194942 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:48.207376 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:48.214552 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:48.215889 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:48.220992 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:48.222738 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:48.325937 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:48.330547 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:48.331120 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:48.331594 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:48.352492 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:48.361721 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:48.366101 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:48.406191 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:48.406870 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:48.414147 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:48.417259 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:48.422907 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:48.424682 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:48.431480 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:48.504679 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:48.515518 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:48.516707 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:48.520161 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:48.524351 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:48.543692 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:48.543998 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:48.544256 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:48.555615 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:48.571219 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:48.582763 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:48.584127 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:48.594120 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:48.602345 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:48.602615 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:48.614904 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:48.616330 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:48.741012 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:48.747032 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:48.749678 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:48.752214 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:48.755175 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:00:48.756542 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:48.770622 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:48.788087 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:48.788594 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:48.791893 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:48.814570 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:48.814687 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:48.815557 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:48.815829 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:48.824162 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:48.948671 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:48.951660 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:48.972667 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:48.974548 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:49.006798 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:49.007908 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:49.011370 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:49.028553 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:49.031908 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:49.037477 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:49.039927 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:49.051513 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:49.051621 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:49.059031 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:49.062151 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:49.063997 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:49.069176 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:49.084132 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:49.100311 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:49.108829 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:49.206264 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:49.210026 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:49.210287 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:49.213733 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:49.216348 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:49.217150 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:49.224985 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:49.274924 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:49.277953 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:49.280256 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:49.280998 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:49.286682 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:49.289379 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:49.302389 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:49.335264 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:49.411988 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:49.412621 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:49.414494 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:49.415509 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:49.418398 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:49.424731 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:49.443578 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:49.490384 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:49.492545 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:49.494805 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:49.496264 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:49.507764 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:49.517027 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:49.522066 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:00:49.545352 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:49.548172 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:49.556210 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:49.558647 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:49.567226 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:49.579403 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:49.672858 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:49.675652 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:49.675743 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:49.676399 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:49.679268 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:49.714996 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:49.720670 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:49.786842 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:49.787560 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:49.790009 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:49.799974 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:49.810140 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:49.813844 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:49.825412 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:49.832405 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:49.850231 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:49.858628 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:49.862231 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:49.866381 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:49.869756 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:49.869940 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:49.872720 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:49.955086 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:49.961434 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:49.968827 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:49.977335 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:49.977573 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:49.979610 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:50.008822 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:50.012998 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:50.051667 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:50.053524 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:50.062847 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:50.065250 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:50.065912 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:50.071600 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:50.083567 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:50.085129 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:50.090577 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:50.095153 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:50.104149 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:50.106272 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:50.126246 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:50.129094 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:50.135176 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:50.234169 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:50.235047 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:50.239010 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:50.243718 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:50.251354 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:50.252677 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:50.319617 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:50.372004 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:50.376039 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:50.377557 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:50.377864 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:50.384964 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:50.388751 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:50.404076 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:50.404282 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:50.407642 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:50.422187 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:50.422595 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:50.422683 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:50.438982 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:50.446841 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:50.535030 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:50.538077 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:50.542912 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:50.543498 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:50.544693 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:50.545508 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:50.546303 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:00:50.577993 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:50.599339 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:00:50.603214 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:50.619485 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:50.621546 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:50.625092 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:50.633416 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:50.639564 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:50.640623 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:50.683475 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:50.697624 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:50.699101 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:50.701750 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:50.717082 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:00:50.717681 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:50.718964 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:50.720079 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:50.752142 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:50.753508 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:00:50.810124 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:00:50.820358 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:50.826723 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:50.832623 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:50.834736 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:50.835580 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:50.842649 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:50.860383 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:50.860417 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:50.907927 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:50.911325 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:50.914068 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:50.917721 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:50.926753 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:50.933472 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:50.940851 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:50.951262 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:00:50.960531 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:00:50.965152 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:50.971053 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:50.977604 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:50.982378 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:50.986485 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:00:50.987156 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:50.992962 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:51.000087 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:51.000442 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:51.048435 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:51.059602 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:51.063748 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:51.064704 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:51.074191 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:51.074492 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:51.141368 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:51.163201 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:51.164517 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:51.172191 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:51.174191 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:51.186019 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:51.201829 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:51.218834 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:51.226866 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:51.227365 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:51.238533 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:51.250301 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:51.256794 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:51.266955 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:51.276139 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:00:51.286695 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:00:51.309737 11598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:00:51.312681 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:51.351631 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:51.352247 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:51.353369 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:51.362812 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:51.365851 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:51.382367 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:51.393982 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:51.398108 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:51.399137 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:51.409760 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:51.416491 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:51.438982 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:51.445061 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:51.457571 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:51.479503 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:51.559319 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:51.566668 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:51.567399 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:51.569741 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:51.573019 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:51.583101 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:51.591398 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:00:51.620302 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:51.622907 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:51.625774 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:51.632324 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:51.635843 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:51.648511 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:51.764734 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:51.777937 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:51.778170 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:51.779284 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:51.783022 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:51.788398 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:51.795022 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:51.797928 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:51.803061 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:51.807152 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:51.815960 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:51.827734 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:51.843371 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:51.847127 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:51.851886 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:51.893288 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:51.897049 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:51.900599 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:51.900857 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:51.908903 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:51.910377 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:00:51.918215 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:51.930697 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:52.021730 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:52.022222 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:52.023342 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:52.035096 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:52.036935 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:52.049496 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:52.055392 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:52.154597 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:52.164677 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:52.165418 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:52.169685 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:52.188346 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:52.196131 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:52.208523 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:52.233750 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:52.243249 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:52.246989 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:52.249617 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:52.254683 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:52.261266 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:52.265583 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:52.348100 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:52.354072 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:52.354300 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:52.369107 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:52.369840 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:52.373229 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:52.383947 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:52.396800 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:52.470614 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:52.472707 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:52.473327 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:52.476595 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:52.476595 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:52.497020 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:52.509276 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:52.527482 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:52.533943 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:52.570195 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:52.577773 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:52.582654 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:52.585873 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:52.591131 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:52.606774 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:52.611191 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:52.673013 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:52.673540 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:52.689167 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:52.691543 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:52.703568 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:52.705374 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:52.719576 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:52.751246 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:52.778046 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:52.782727 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:52.785164 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:52.793130 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:00:52.807307 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:52.809821 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:52.826504 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:52.862515 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:52.864445 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:52.869026 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:52.873275 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:52.880659 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:52.882463 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:52.882493 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:52.936671 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:52.937130 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:52.937839 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:52.937983 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:52.940930 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:52.943154 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:52.951867 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:52.972297 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:00:53.015004 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:53.054399 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:53.060166 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:53.066257 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:53.071223 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:53.073774 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:53.075903 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:53.091888 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:53.130287 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:53.142755 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:53.143030 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:53.148610 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:53.149902 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:53.151158 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:53.177253 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:53.200371 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:53.241153 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:53.241575 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:53.246347 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:53.248574 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:53.259706 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:53.264166 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:53.267119 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:00:53.274554 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:00:53.284475 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:00:53.319237 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:53.330585 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:53.340255 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:53.340269 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:53.341823 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:53.355981 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:53.359472 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:53.379397 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:53.389230 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:00:53.394001 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:00:53.457720 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:53.476549 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:53.477721 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:53.483875 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:53.489208 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:53.489753 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:53.490554 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:53.522231 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:53.532325 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:53.533071 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:53.541296 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:53.563575 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:53.573812 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:53.587987 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:00:53.628147 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:53.683280 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:53.687299 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:53.693118 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:53.705878 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:53.707396 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:53.742067 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:53.743172 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:53.751552 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:53.753274 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:53.755738 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:53.765902 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:53.778899 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:53.786540 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:53.795451 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:53.800429 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:53.802171 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:53.805022 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:53.815251 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:53.826226 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:53.826674 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:53.830247 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:53.857858 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:00:53.920756 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:53.931360 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:53.934729 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:53.936718 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:53.938535 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:53.946731 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:53.950016 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:53.962260 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:54.058612 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:54.059230 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:54.062534 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:54.073007 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:54.087093 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:54.100989 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:54.109724 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:54.117158 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:54.127746 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:54.135681 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:54.143450 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:54.156342 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:54.160811 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:54.208822 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:54.223645 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:54.227418 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:54.234663 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:54.237290 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:54.247400 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:54.257966 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:54.333618 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:54.340260 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:54.345504 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:54.350500 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:54.356653 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:54.362655 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:54.362890 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:54.363919 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:54.365556 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:54.393026 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:54.395248 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:54.397591 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:54.400749 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:54.421387 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:54.432806 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:00:54.466684 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:00:54.514143 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:54.521050 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:54.537593 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:54.540621 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:54.541348 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:54.552188 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:54.556278 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:54.594643 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:54.594662 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:54.594758 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:54.598501 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:54.612077 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:54.682997 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:54.730679 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:54.744454 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:54.749683 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:54.756223 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:54.757771 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:54.758626 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:54.761178 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:54.769487 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:54.771867 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:54.808558 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:54.811452 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:54.812196 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:54.816738 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:54.824218 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:54.836145 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:54.847333 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:54.848351 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:54.853757 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:54.867048 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:54.871346 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:54.940675 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:54.969307 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:54.974336 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:54.975876 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:54.987129 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:54.989040 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:54.989954 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:54.991458 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:54.998111 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:55.039108 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:55.043325 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:55.044262 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:55.052181 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:55.052938 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:55.054208 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:55.058305 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:55.131599 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:55.173804 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:55.174008 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:55.182029 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:55.182330 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:55.186640 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:55.204722 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:55.212980 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:55.219462 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:55.221057 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:55.247411 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:55.247411 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:55.247581 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:55.248385 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:55.255378 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:00:55.273087 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:00:55.280908 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:00:55.301824 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:55.384213 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:55.385200 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:55.386399 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:55.391736 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:55.398730 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:55.413568 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:55.418025 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:55.428813 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:55.430634 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:55.431938 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:55.432067 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:55.471822 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:55.475466 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:55.523867 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:55.538590 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:55.577589 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:55.581306 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:55.582855 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:55.588431 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:55.600283 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:55.601401 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:55.603789 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:55.617175 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:55.617442 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:55.645612 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:55.650192 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:55.650897 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:55.652186 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:55.733315 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:00:55.772068 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:00:55.774915 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:55.781208 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:55.782109 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:55.782629 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:55.784044 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:55.785890 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:55.791771 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:55.791939 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:55.793624 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:55.826524 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:55.829569 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:55.844753 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:55.846302 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:55.848578 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:55.849064 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:55.851680 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:55.883709 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:00:55.884091 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:55.980690 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:55.981678 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:55.996647 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:56.000751 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 08:00:56.002560 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:56.004153 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:56.006108 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:56.019918 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:56.034579 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:56.036195 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:56.053185 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:56.057397 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:56.057723 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:56.060230 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:56.100219 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:56.101589 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:56.107228 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:56.107497 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:56.109431 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:56.123544 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:56.137920 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:56.224299 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:56.227727 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:56.227861 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:56.228932 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:56.234333 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:56.248811 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:56.265031 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:00:56.273887 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:56.295822 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:56.296109 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:56.301400 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:56.307986 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:56.310897 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:56.318637 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:56.325531 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:00:56.387162 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 08:00:56.387991 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:56.403841 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:00:56.404229 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:00:56.420377 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:56.458114 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:56.460489 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:56.468899 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:56.478658 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:56.482261 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:56.486546 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:56.488032 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:56.501348 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:56.516019 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:56.523908 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:56.524518 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:56.526588 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:56.536191 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:56.560400 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:56.567816 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:56.571831 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:56.572973 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:56.586629 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:56.630060 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:00:56.648659 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:00:56.702876 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:56.707123 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:56.722384 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:56.727344 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:56.734922 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:56.739557 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:56.751510 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:56.765436 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:56.772118 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:56.778409 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:56.780661 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:56.782665 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:56.803714 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:56.804244 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:56.812366 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:56.897655 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:56.900348 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:56.900553 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:56.902004 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:56.921391 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:56.926805 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:56.946573 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:00:56.956851 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:56.983368 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:56.989385 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:56.998558 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:57.002614 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:57.005105 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:57.005172 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:57.010886 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:57.013669 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:00:57.016688 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:57.024219 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:57.024853 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:57.030627 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:57.153482 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:57.156028 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:57.157058 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:57.158203 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:57.161039 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:57.166640 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:57.177408 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:57.239300 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:57.277046 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:57.277101 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:57.278710 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:57.279193 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:57.289249 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:57.302618 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:57.328220 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:00:57.384897 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:00:57.412457 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:57.420033 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:57.424026 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:57.429738 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:57.444960 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:00:57.445233 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:00:57.449463 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:57.449447 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:57.452782 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:57.467162 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:57.469987 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:00:57.480289 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:57.487488 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:57.492460 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:57.518035 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:00:57.575242 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:00:57.588825 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:57.589536 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:57.599485 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:57.605175 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:57.609659 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:57.641772 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:57.652194 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:00:57.657492 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:57.662990 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:57.663672 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:57.667517 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:57.669720 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:57.698730 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:57.711503 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:00:57.776471 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:00:57.780014 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:00:57.781766 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:57.794781 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:57.799971 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:57.809149 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:57.817538 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:57.830669 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:00:57.844111 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:57.849668 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:57.852003 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:57.859010 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:57.859660 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:57.860227 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:57.874217 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:57.891009 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:57.899295 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:57.920723 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:57.930619 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:57.934697 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:57.940178 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:57.959955 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:57.962585 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:58.047159 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:58.048758 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:58.051107 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:58.060889 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:58.061558 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:58.065198 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:58.094483 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:00:58.099169 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:58.102319 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:58.115576 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:58.124811 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:58.127478 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:58.136790 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:58.207144 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:00:58.213773 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:00:58.221820 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:00:58.227243 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:58.264806 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:58.279438 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:58.282191 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:58.286273 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:58.297935 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:00:58.299652 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:58.325656 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:58.356545 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:00:58.370633 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:00:58.389580 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:58.398861 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:58.403192 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:00:58.412227 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:58.416551 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:58.430931 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:58.433890 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:00:58.434942 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:00:58.475518 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:00:58.482844 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:58.483724 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:58.484140 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:58.484433 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:58.489316 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:58.524744 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:58.535485 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:00:58.589474 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:58.592624 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:58.594628 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:58.594648 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:58.595896 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:58.604358 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:58.618067 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:00:58.657223 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:58.660909 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:58.660979 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:58.668176 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:58.676622 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:58.686158 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:00:58.695920 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:00:58.736789 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:58.764107 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:58.764215 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:58.771725 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:58.772421 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:58.774813 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:58.787935 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:58.848109 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:58.863379 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:58.868289 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:58.869226 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:58.874899 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:58.887770 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:58.895835 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:58.930778 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:00:58.960558 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:58.967263 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:58.973948 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:58.978397 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:58.978633 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:58.980392 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:58.985518 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:59.015502 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:59.035722 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:59.037170 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:59.043658 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:59.060228 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:59.068051 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:00:59.187975 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:00:59.206384 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:59.206442 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:59.207335 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:59.211432 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:59.212673 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:59.225147 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:59.279440 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:00:59.337553 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:59.343186 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:00:59.347427 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:59.355022 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:59.364085 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:59.365658 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:59.377807 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:59.392897 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:00:59.437510 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:00:59.441910 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:00:59.448580 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:59.455029 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:59.455842 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:59.455883 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:59.461678 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:59.466900 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:59.469616 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:59.474786 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:59.477869 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:59.488927 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:59.490624 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:59.511665 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:59.592586 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:00:59.594633 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:59.595127 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:59.600965 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:59.607378 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:59.607839 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:59.611827 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:59.679826 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:59.681300 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:59.692313 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:59.694972 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:59.696422 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:59.699119 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:59.778488 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:00:59.781984 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:00:59.795166 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:59.803064 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:59.811442 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:59.811749 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:59.816368 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:00:59.827013 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:00:59.859963 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:00:59.896853 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:59.897905 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:59.908552 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:59.930585 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:59.934055 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:59.937623 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:00:59.950034 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:00:59.951064 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:00:59.959996 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:00:59.969971 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:00:59.972205 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:00:59.983525 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:00.037089 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:00.039281 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:00.048370 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:00.058951 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:00.062130 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:01:00.065049 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:00.067116 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:00.084755 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:00.098666 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:00.118865 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:01:00.137979 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:00.142345 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:00.143826 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:00.144560 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:00.148478 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:00.154654 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:00.164125 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:00.179870 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:00.199246 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:01:00.234401 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:00.238032 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:00.246277 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:00.247638 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:00.251246 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:00.262732 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:00.284919 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:00.304818 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:01:00.352445 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:00.357587 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:00.358350 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:00.383057 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:00.384529 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:00.388013 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:00.405777 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:00.407363 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:00.407814 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:00.407943 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:00.408852 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:00.438691 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:00.442641 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:00.458290 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:00.469247 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:01:00.524560 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:00.529141 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:00.529392 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:00.530508 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:00.544809 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:00.561988 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:00.568447 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:01:00.573879 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:00.594642 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:00.594834 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:01:00.603474 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:00.605027 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:00.609576 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:00.611829 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:00.653151 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:00.681815 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:00.693750 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:00.715873 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:00.735311 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:00.735754 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:00.738794 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:00.750331 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:00.753726 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:00.770146 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:00.770519 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:00.783565 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:00.786262 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:00.786716 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:00.788671 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:00.821334 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:00.827847 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:00.839205 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 08:01:00.872087 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:00.872294 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:00.872799 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:00.873041 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:00.882026 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:01:00.888806 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:00.907685 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:00.938403 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:00.964453 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:00.967489 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:01.064623 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:01:01.124531 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:01.130628 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:01.131263 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:01.140750 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:01.140802 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:01.149817 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:01.154661 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:01:01.159232 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:01.191838 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:01.195351 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:01.208220 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:01.211961 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:01.221902 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:01.224581 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:01.225077 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:01:01.299712 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:01.302772 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:01.306324 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:01.311159 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:01.312975 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:01.321847 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:01.360903 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 08:01:01.390009 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:01.392349 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:01.392838 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:01.410965 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:01.421811 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:01.425271 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:01.429898 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:01.435056 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:01.438923 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:01.446878 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:01.464911 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:01.467314 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:01.475929 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:01.489976 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:01.552174 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:01:01.560050 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:01:01.571445 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:01.580668 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:01.583361 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:01.603883 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:01.607277 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:01.608687 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:01.623998 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:01.628149 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:01.646683 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:01.646783 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:01.648490 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:01.653700 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:01.706254 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:01.745016 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:01.763543 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:01.770057 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:01.771947 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:01.780367 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:01.784081 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:01:01.792346 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:01.793298 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:01.803598 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:01.805258 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:01.816901 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:01.817749 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:01.842131 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:01.855010 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:01.908721 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:01.909472 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:01.910326 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:01.911940 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:01.912844 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:01.918603 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:01.988320 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:01.991053 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:01.992798 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:01.994571 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:02.002806 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:01:02.009203 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:02.021165 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:02.028997 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:01:02.042177 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:01:02.043463 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:02.049522 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:02.059032 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:01:02.079576 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:02.082301 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:02.082379 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:02.083869 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:02.086165 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:02.112784 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:02.183257 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:02.189958 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:02.194286 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:02.194845 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:02.199437 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:02.221613 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:02.243511 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:02.270254 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:02.272751 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:02.274252 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:02.274768 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:02.283119 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:01:02.288800 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:02.307390 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:02.313248 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:02.327724 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:02.329149 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:02.340001 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:01:02.341823 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:02.350826 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:02.373246 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:02.428059 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:01:02.444581 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:02.449528 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:02.451289 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:02.479455 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:02.480492 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:02.484817 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:02.513299 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:01:02.518018 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:02.518677 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:02.538924 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:02.543555 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:02.549127 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:02.550498 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:02.581633 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:02.606927 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:02.632242 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:02.637111 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:02.640153 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:01:02.642681 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:02.652048 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:02.652048 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:02.655527 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:02.724174 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:02.727720 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:02.727871 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:02.730521 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:02.740113 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:02.740483 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:02.759785 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:02.820668 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:02.829124 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:02.829432 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:02.834321 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:01:02.852102 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:02.854449 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:02.855772 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:02.857946 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:02.861239 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:01:02.866873 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:02.880657 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:02.969756 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:02.974443 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:02.976344 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:02.985450 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:02.991928 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:02.995407 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:03.001046 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:03.004717 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:01:03.008818 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:03.015152 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:03.024400 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:03.034754 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:03.035909 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:01:03.045776 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:03.053680 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:03.079206 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:03.165030 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:01:03.165289 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:03.167213 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:03.172204 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:03.176399 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:03.176699 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:03.191998 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:03.231163 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:03.239279 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:03.239758 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:03.240223 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:03.240469 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:03.242380 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:03.243324 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:01:03.259347 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:03.265250 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:03.271955 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:03.281262 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:03.289130 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:03.294411 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:03.320438 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:03.321493 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:03.426968 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:03.436502 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:03.440908 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:03.451678 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:03.453959 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:03.457000 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:03.466668 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:03.479611 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:03.481102 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:03.486579 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:03.498538 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:03.503463 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:03.506609 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:01:03.511474 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:03.522610 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:03.589180 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:01:03.596668 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:03.603055 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:03.604925 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:03.608270 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:03.611680 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:03.619913 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:03.621454 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:01:03.699354 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:03.699523 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:03.700078 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:03.706668 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:03.708647 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:03.710860 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:03.711668 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:03.713881 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:03.718840 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:03.734511 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:03.740116 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:03.746919 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:03.773658 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:03.825667 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:03.881321 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:03.883127 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:03.888984 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:03.893056 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:03.896138 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:03.906643 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:03.910657 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:03.917032 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:03.925738 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:03.940750 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:03.942504 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:03.942957 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:03.981187 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:04.009460 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:01:04.043470 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:04.044651 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:04.047199 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:04.064455 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:04.082178 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:04.090351 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:04.094630 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:04.127527 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:04.132650 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:04.136041 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:04.154546 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:04.161213 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:04.166944 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:04.178778 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:04.181293 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:04.194001 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:04.209568 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:04.210316 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:04.224779 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:04.256640 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:04.302582 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:04.303890 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:04.304100 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:04.312263 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:04.314608 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:04.318727 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:04.329840 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:04.351368 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:01:04.357925 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:01:04.447587 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:04.452162 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:04.454166 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:04.457958 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:04.476292 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:04.482772 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:04.483188 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:01:04.488097 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:04.489516 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:04.489581 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:01:04.508250 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:04.509642 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:04.521248 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:04.533071 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:04.568567 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:04.604008 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:04.604760 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:04.607645 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:04.607702 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:04.609171 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:04.625190 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:04.659410 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:04.672482 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:04.702792 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:04.716647 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:04.719311 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:04.724980 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:04.726456 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:04.736485 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:04.738535 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:04.839527 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:04.855731 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:04.863049 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:04.871860 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:04.894510 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:04.895164 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:04.909425 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:04.909499 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:04.916379 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:04.917940 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:04.919483 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:04.935353 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:04.937379 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:04.943073 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:04.961786 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:04.967471 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:04.972414 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:04.987039 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:04.991083 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:04.993564 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:04.995678 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:04.996259 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:05.013654 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:05.090057 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:05.103292 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:05.105326 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:05.116986 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:05.127140 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:05.128372 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:05.138729 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:05.173151 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:05.176020 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:05.179778 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:05.182410 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:05.200562 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:05.200841 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:05.250871 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:05.255673 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:05.290930 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:05.300582 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:05.302824 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:05.312621 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:05.319471 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:05.321622 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:05.340588 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:05.354740 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:05.363735 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:05.368592 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:05.377640 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:05.381099 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:05.391119 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:05.396638 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:05.403306 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:05.423491 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:05.424706 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:05.429134 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:05.431861 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:05.433308 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:05.434389 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:05.437327 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:05.459117 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:05.555881 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:05.556984 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:05.560604 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:05.571569 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:05.578212 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:05.586629 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:05.617650 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:05.618165 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:05.622335 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:05.622805 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:05.626998 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:05.631127 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:05.638268 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:05.755684 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:05.756947 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:05.760741 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:05.761158 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:05.761914 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:05.765476 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:05.780217 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:05.783206 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:05.807914 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:05.843717 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:05.848472 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:05.855093 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:05.857933 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:05.875926 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:05.876155 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:05.877702 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:05.886617 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:05.886960 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:05.887894 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:05.895903 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240501 08:01:05.907819 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:05.918115 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:05.923599 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:05.941231 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:05.943032 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:05.977931 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:05.984774 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:06.007506 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:06.038661 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:06.041455 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:06.044114 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:06.046389 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:06.048877 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:06.074312 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:06.145237 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:06.148483 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:06.151234 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:06.152572 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:06.167390 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:01:06.176057 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:01:06.183593 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:06.183684 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:06.262847 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:06.263304 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:06.275297 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:06.277830 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:06.289791 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:06.293560 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:06.297021 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:06.329530 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:06.334633 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:06.348038 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:06.357697 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:06.374671 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:06.386168 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:06.388023 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 83 similar messages]
I20240501 08:01:06.404291 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:06.408874 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:06.414101 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:06.420897 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:06.423818 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:06.425714 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:06.426154 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:06.505307 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:06.530004 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:06.530467 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:06.533277 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:06.540741 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:06.547269 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:06.549405 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:06.566771 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:06.589438 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:06.598368 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:06.598469 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:06.614255 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:06.623342 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:06.623401 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:06.628051 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 08:01:06.679081 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:06.697528 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:06.710984 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:06.722362 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:06.724578 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:06.730406 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:06.733186 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:06.741619 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:06.794580 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:06.795666 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:06.796934 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:06.799077 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:06.800384 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:06.811650 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:06.820905 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:06.823459 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:06.837217 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:06.915431 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:06.916415 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:06.917119 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:06.926106 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:06.926218 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:06.928746 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:06.942189 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:06.970763 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:06.990093 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:06.995699 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:06.996234 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:07.001806 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:07.004832 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:07.005054 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:07.022948 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:07.085302 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:01:07.097878 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:07.108424 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:07.109872 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:07.123248 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:07.123422 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:07.131703 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:07.146853 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:07.148998 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:07.160508 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:07.167088 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:07.172405 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:07.182612 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:07.216925 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:07.219959 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:07.230859 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:07.231422 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:07.233788 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:07.234575 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:07.240224 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:07.260684 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:07.364969 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:07.375322 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:07.376559 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:07.386847 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:07.387501 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:07.389365 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:07.416200 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:07.432783 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:07.433363 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:07.440941 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:07.445230 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:07.447204 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:07.447270 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:07.459645 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:01:07.462924 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:07.492197 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:07.548671 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:07.550300 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:07.556871 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:07.560986 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:07.562109 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:07.564033 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:07.574025 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:07.619287 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:07.623931 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:07.628086 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:07.633206 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:07.635294 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:07.638473 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:07.645332 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:07.645664 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:07.647444 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:07.659725 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:07.696282 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:07.741324 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:07.747654 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:07.750627 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:07.752126 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:07.752347 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:07.769686 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:07.769922 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:07.802403 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:07.822588 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:07.827291 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:07.828795 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:07.830188 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:07.830722 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:07.834632 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:07.938484 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:07.947343 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:07.949220 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:07.949975 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:07.957173 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:07.957253 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:07.978760 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:07.982137 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:08.010543 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:08.012754 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:08.021162 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:01:08.022889 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:08.024394 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:08.041034 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:08.047509 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:08.069389 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:08.084365 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:08.089545 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:08.091008 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:08.092800 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:08.094184 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:08.099421 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:08.116493 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:08.146564 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:08.194730 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:08.197541 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:08.204185 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:08.206359 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:08.214187 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:08.214304 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:08.267740 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:08.268159 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:08.273772 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:08.276662 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:08.294042 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:08.294617 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:08.332680 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:08.351580 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:08.380657 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:08.410291 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:08.418450 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:08.419000 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:08.430483 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:08.439669 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:08.447202 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:08.459304 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:08.468632 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:08.475015 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:08.479559 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:08.481611 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:08.482494 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:08.495494 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:08.514616 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:08.514971 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:08.519001 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:08.539302 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:08.543076 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:08.543963 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:08.544389 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:08.545768 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:08.547268 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:08.571224 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:08.581291 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:08.655906 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:08.656522 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:08.657868 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:08.658222 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:08.668226 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:08.687536 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:01:08.695649 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:08.793013 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:08.793889 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:08.798117 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:08.798163 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:08.799226 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:08.819375 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:08.871021 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:08.872467 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:08.873625 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:08.876848 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:08.888672 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:08.894724 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:08.950006 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:08.959372 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:08.960600 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:08.963534 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:08.966261 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:08.968984 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:08.978709 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:08.982633 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:09.048983 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:09.064339 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:09.064666 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:09.079859 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:09.080065 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:09.090579 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:09.106746 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:09.130884 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:09.151247 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:09.151279 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:09.160420 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:09.165464 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:09.167596 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:09.170238 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:09.194579 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:09.266731 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:01:09.296213 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:09.299811 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:09.300084 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:09.305421 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:09.316298 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:09.352569 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:09.357764 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:01:09.358655 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:09.363713 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:09.368950 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:09.383977 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:09.391109 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:09.393636 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:09.402879 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:09.502916 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:09.508548 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:09.509351 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:09.510737 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:09.513939 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:09.515405 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:09.518904 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:09.524629 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:09.543417 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:09.544262 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:09.552778 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:09.565712 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:01:09.570616 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:09.574826 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:09.594853 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:09.630311 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:01:09.688776 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:01:09.693833 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:09.696494 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:09.696754 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:09.704810 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:09.713923 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:09.733211 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:09.769359 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:09.770990 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:09.772271 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:09.773115 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:09.780036 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:09.789095 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:09.832655 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:09.836041 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:09.838270 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:09.853542 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:09.857702 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:01:09.861176 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:09.861181 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:09.925643 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:09.948957 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:01:09.971428 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:09.973793 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:09.974946 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:09.977936 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:09.996855 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:10.004703 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:10.024613 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:10.025215 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:10.025523 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:10.026187 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:10.033663 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:10.049412 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:10.112665 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:10.124662 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:10.134971 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:10.155954 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:01:10.166215 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:10.170940 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:10.181582 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:10.188072 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:01:10.188659 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:10.189726 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:10.191226 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:01:10.204675 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:10.239261 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:10.246074 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:10.248665 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:10.253785 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:10.265393 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:10.270582 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:10.303560 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:10.311354 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:10.314615 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:10.324117 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:10.335865 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:10.338351 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:10.365764 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:10.431623 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:01:10.437562 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:10.439435 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:10.439456 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:10.446954 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:10.457028 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:10.463336 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:10.493435 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:01:10.498755 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:10.503672 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:10.504161 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:10.504348 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:10.506199 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:10.514441 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:10.617568 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:10.627261 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:10.632339 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:10.634461 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:10.637555 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:10.647701 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:10.652245 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:10.657366 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:10.685129 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:10.691651 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:01:10.709091 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:10.714533 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:10.716202 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:10.724810 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:10.746083 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:10.747105 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:10.749433 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:10.816134 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:10.828231 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:01:10.835592 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:10.836946 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:10.855367 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:10.855581 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:10.860445 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:10.861569 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:10.890491 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:10.893289 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:01:10.903251 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:10.913771 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:10.922372 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:10.924563 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:10.936618 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:10.947813 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:01:10.957571 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:11.000768 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:11.007686 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:11.009821 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:11.010607 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:11.011282 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:11.015267 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:11.016213 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:11.050166 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:11.058486 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:11.091495 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:11.106343 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:11.115180 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:11.115621 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:11.123950 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:11.124918 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:11.140163 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:11.171551 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:11.181852 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:01:11.211637 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:11.214210 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:11.216790 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:11.217304 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:11.218139 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:11.225975 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:11.299157 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:11.299340 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:11.306108 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:11.317137 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:11.321379 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:11.332490 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:11.414407 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:11.414410 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:11.418851 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:11.423012 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:11.426528 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:11.426604 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:11.443635 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:11.516634 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:11.516957 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:11.520648 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:11.520908 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:01:11.528636 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:11.536481 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:11.540254 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:11.542312 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:11.549110 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:01:11.558885 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:11.567989 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:11.568642 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:11.576710 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:11.581544 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:11.594748 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:11.634483 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:01:11.651623 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:11.651731 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:11.663450 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:11.671692 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:11.676949 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:11.677138 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:11.677213 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:11.691679 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:11.720718 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:01:11.756605 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:11.758857 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:11.765524 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:11.768903 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 08:01:11.778246 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:11.780992 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:11.786674 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:11.881001 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:11.883734 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:11.887230 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:11.900750 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:11.914134 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:11.915582 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:01:11.922922 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:11.969547 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:11.971309 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:11.972097 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:11.975888 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:11.978978 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:11.986410 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:11.988400 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:11.989126 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:11.992575 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:11.995273 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:12.014917 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:12.050658 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:12.059402 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:12.070897 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:12.082118 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:12.121467 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:12.125392 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:12.126096 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:12.150803 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:12.163245 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:12.163690 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:12.184686 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:01:12.219964 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:12.235630 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:12.236781 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:12.237560 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:12.239904 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:12.253857 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:12.270143 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:12.332052 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:12.389417 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:12.396590 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:12.398698 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:12.403615 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:12.415725 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:12.424197 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:12.427589 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:12.429281 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:12.436270 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:01:12.452626 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:12.453065 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:12.471297 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:12.479955 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:12.483345 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:12.485294 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:12.486915 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:12.493086 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:12.498347 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:12.517292 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:12.523115 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:12.578408 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:12.616042 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 08:01:12.630456 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:12.649065 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:12.650297 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:12.656420 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:12.657495 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:12.661684 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:12.670717 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:12.679143 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:12.700055 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:12.700237 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:12.728123 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:12.800554 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:12.801270 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:12.802151 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:12.815279 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:12.817750 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:12.824649 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:12.837586 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:12.837875 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:12.895224 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:12.895236 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:12.900563 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:12.914132 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:12.921546 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:12.929549 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:13.000806 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:13.005115 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:13.005811 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:13.012058 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:13.041361 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:13.045109 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:13.059769 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:13.062538 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:13.064529 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:13.067914 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:13.070542 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:13.075095 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:13.087553 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:13.088307 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:13.093549 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:13.133808 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:13.148209 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:13.167002 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:13.172515 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:13.201222 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:13.225275 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:13.230132 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:13.235453 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:13.236848 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:13.238917 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:13.247256 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:13.250597 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:13.260603 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:13.266705 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:13.275676 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:13.283344 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:13.306030 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:13.363812 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:13.376933 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:13.378376 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:13.378376 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:13.385107 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:01:13.385862 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:13.393685 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:13.408146 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:13.414075 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:13.453127 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:13.461359 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:13.464962 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:13.466876 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:13.472100 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:13.481462 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:13.487490 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:13.523128 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:13.526368 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:13.529965 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:13.530314 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:13.537807 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:13.538841 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:13.546553 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:13.562129 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:13.650566 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:13.656452 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:13.658099 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:13.658535 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:13.671595 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:13.689220 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:13.689555 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:13.750751 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:13.751284 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:13.752740 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:13.753098 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:13.754099 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:13.756145 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:13.758152 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:13.847319 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:13.850992 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:13.859146 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:13.870213 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:13.880965 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:13.908774 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:13.919282 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:13.928757 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:13.932794 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:13.933580 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:13.947899 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:13.949709 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:13.969753 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:13.972316 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:13.976918 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:01:13.995863 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:13.996482 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:13.998683 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:14.007978 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:14.008210 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:14.035347 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:14.077870 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:14.123924 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:14.124718 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:14.125310 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:14.128988 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:14.134622 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:14.137656 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:14.154758 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:14.191388 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:14.196995 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:14.203298 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:14.203444 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:14.207109 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:14.211100 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:14.220144 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:14.288856 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:14.341825 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:14.342804 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:14.346184 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:14.348516 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:14.352053 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:14.363600 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:14.398084 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:14.399092 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:14.414657 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:14.418865 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:14.423696 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:14.430585 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:14.439718 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:14.444193 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:14.457620 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:14.460110 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:01:14.461104 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:14.463176 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:14.474887 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:14.479799 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:14.507668 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:14.570262 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:14.741888 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:14.776428 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:01:14.817773 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:14.819612 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:14.827219 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:01:14.833065 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:14.835026 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:14.843230 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:14.865396 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:01:14.871872 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:14.875120 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:14.875434 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:14.877106 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:14.881811 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:14.905930 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:14.908260 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:14.941242 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:14.944134 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:14.946478 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:14.966070 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:14.967339 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:14.969398 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:14.986449 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:14.992069 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:01:14.995996 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:01:15.075510 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:15.076707 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:15.083789 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:15.083971 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:15.090669 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:15.099161 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:15.113716 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:15.136658 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:15.149279 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:15.157764 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:01:15.157953 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:15.163329 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:01:15.163959 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:15.175804 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:15.177214 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:15.257236 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:15.299911 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:01:15.329720 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:15.334473 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:15.336369 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:15.359674 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:15.365116 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:15.365761 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:01:15.366412 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:15.374879 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:15.382882 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:15.393476 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:01:15.393551 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:15.398561 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:15.415473 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:15.429688 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:15.439620 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:15.452468 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:01:15.485821 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:15.490281 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:15.503118 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:15.504722 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:15.509091 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:15.512509 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:15.524878 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:15.556155 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:15.588654 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:15.619802 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:15.619854 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:15.626340 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:15.635916 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:15.640755 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:15.657425 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:15.663897 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:15.710642 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:01:15.716869 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:15.719241 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:15.722993 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:15.726284 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:15.750542 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:15.771723 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:15.821940 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:15.824491 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:15.833444 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:15.835770 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:15.846901 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:15.847316 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:15.881121 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:01:15.892773 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:15.895133 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:15.895341 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:15.908879 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:15.914196 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:15.922509 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:15.935330 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:15.961622 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:01:15.979167 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:15.979318 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:15.981318 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:15.985070 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:15.985070 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:15.987356 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:16.008067 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:16.069475 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 08:01:16.081574 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:16.084784 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:16.104944 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:16.111157 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:16.121284 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:16.124451 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:16.125191 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:16.128885 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:16.138973 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:16.156939 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:16.157922 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:01:16.172983 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:16.173727 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:16.185822 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:16.185971 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:16.268427 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:01:16.297453 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:16.301055 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:16.307719 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:16.316994 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:16.326995 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:16.332965 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:16.360189 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:16.365841 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:16.366003 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:16.369192 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:16.381071 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:16.385277 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:16.393647 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:16.395004 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:16.400204 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:16.409485 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:16.412590 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:16.419801 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:16.424783 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:16.442587 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:16.541088 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:16.541462 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:16.548422 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:16.550005 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:16.557221 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:16.558956 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:16.581909 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:16.675208 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:16.684639 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:16.732393 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:01:16.745972 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:16.750079 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:16.752357 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:16.766157 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:16.777487 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:16.790845 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:16.836881 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:16.844456 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:16.848984 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:16.848999 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:16.851922 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:01:16.859182 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:16.870752 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:16.873284 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240501 08:01:16.874538 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:16.877941 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:16.922628 11598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 08:01:16.977679 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:16.984247 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:16.984289 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:16.985803 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:17.000746 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:17.003839 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:17.012203 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:17.015040 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:17.017405 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:17.029557 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:17.031288 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:17.048089 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:17.064930 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:01:17.175498 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:17.187975 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:17.188481 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:17.195065 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:17.200738 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:17.212591 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:17.230764 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:17.230904 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:17.232234 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:01:17.234922 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:17.238178 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:17.271050 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:17.272156 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:17.273236 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:17.283123 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:01:17.284042 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:17.286114 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:17.296154 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:17.304370 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:17.305367 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:17.308362 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:17.323535 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:17.369891 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:01:17.416394 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:17.418161 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:17.421502 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:17.439287 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:17.441187 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:17.462738 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:17.476936 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:17.481855 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:01:17.485226 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:17.486603 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:01:17.496155 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:17.496192 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:17.519496 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:17.521695 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:17.526939 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:01:17.539580 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:17.590714 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:01:17.612250 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:17.620360 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:17.622504 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:17.625229 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:17.631322 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:17.633219 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:17.659619 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:01:17.689041 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:17.695580 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:17.696234 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:17.701165 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:17.710177 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:17.725622 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:17.758093 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:17.766662 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:17.779860 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:17.787415 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:17.794711 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:17.797756 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:17.806056 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:17.806314 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:17.882200 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:17.882484 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:01:17.895215 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:17.895908 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:01:17.897338 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:17.902307 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:17.909862 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:17.915493 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:17.923455 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:17.943499 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:17.945629 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:01:17.947849 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:01:17.948618 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:17.949371 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:17.951438 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:17.963867 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:17.986400 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:18.007122 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:18.070838 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:18.078656 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:18.083593 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:18.090390 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:18.091410 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:18.100601 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:18.128144 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:01:18.144227 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:18.149104 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:18.149483 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:18.150490 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:18.168152 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:18.168792 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:18.186960 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:18.194718 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:18.254637 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:18.254752 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:18.256778 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:18.259665 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:18.263314 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:18.263605 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:18.320605 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:18.348965 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:18.352161 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:01:18.352874 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:18.353932 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:18.356745 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:18.357750 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:18.358892 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:18.470340 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:01:18.490607 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:18.492585 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:18.492969 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:18.511691 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:18.517671 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:18.519518 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:18.523258 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:18.525561 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:18.527480 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:01:18.528210 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:01:18.528641 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:18.559078 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:18.561448 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:18.570196 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:18.573792 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:18.579902 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:01:18.669450 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:18.673902 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:18.674297 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:18.676626 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:18.682667 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:18.696080 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:01:18.701134 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:18.725821 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:18.745270 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:18.745507 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:18.759052 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:18.762063 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:18.766196 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:18.766352 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:18.777801 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:18.783470 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:18.895011 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:18.907617 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:18.912267 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:18.914027 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:18.914522 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:18.917001 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:18.925014 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:18.927913 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:18.931317 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:18.952551 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:18.957326 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:18.972537 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:19.060453 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:19.085547 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:19.095007 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:01:19.102686 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:19.107364 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:19.109074 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:19.122462 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:19.133220 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:19.147454 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:19.159690 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:19.163012 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:19.185284 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:19.187340 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:19.198680 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:19.211110 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:19.214979 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:19.218958 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:19.222522 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:19.223726 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:19.232707 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:19.232836 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:19.251178 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:19.264362 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:19.356170 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:19.396250 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:19.404726 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:19.432372 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:19.435284 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:19.438578 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:19.444764 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:19.450650 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:19.454361 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:19.482477 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:19.490311 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:19.490793 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:19.492895 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:19.503181 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:19.527918 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:19.542340 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:19.640838 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:19.647415 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:19.656090 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:19.658751 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:19.671973 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:19.672888 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:19.674324 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:19.674881 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:19.681605 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:19.704717 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:19.704900 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:19.725126 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:19.732296 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:19.742902 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:19.758673 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:19.770721 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:19.771234 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:19.773618 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:19.776875 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:19.793329 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:19.796214 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:19.858726 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:19.911114 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:19.911386 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:19.913954 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:19.921288 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:19.928005 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:19.945066 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:19.947610 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:19.947906 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:19.956385 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:19.957767 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:19.960896 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:19.984674 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:19.988201 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:19.991745 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:20.014276 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:20.014729 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:20.044644 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:01:20.064141 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:20.100658 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:20.104915 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:20.105094 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:20.106601 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:20.108245 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:20.117902 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:20.129827 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:20.133515 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:20.179584 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:20.182885 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:20.183190 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:20.187197 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:01:20.194804 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:20.195183 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:20.199313 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:20.303411 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:20.312680 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:20.312968 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:20.313300 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:20.320161 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:20.340401 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:20.349412 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:20.368777 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:20.373502 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:20.377429 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:20.382656 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:20.385506 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:20.399423 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:20.404801 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:20.411607 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:20.455271 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:20.466375 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:20.483512 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:20.489351 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:20.495035 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:20.498495 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:20.511368 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:20.523474 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:20.568953 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:20.569801 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:20.570370 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:20.579583 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:20.580315 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:20.599987 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:20.614756 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:20.636700 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:20.644456 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:20.646749 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:20.647678 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:20.648761 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:20.650072 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:20.670585 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:20.686051 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:20.747056 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:20.750372 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:20.751492 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:20.752475 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:20.754866 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:20.780627 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:20.885761 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:20.886468 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:20.891311 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:20.894762 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:20.904990 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:20.905123 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:20.907894 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:20.939164 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:20.943614 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:20.959615 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:20.978760 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:20.983863 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:20.983928 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:20.987926 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:20.996201 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:21.008095 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:21.027753 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:21.078672 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:21.084673 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:21.085822 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 83 similar messages]
I20240501 08:01:21.088356 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:21.091745 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:21.092818 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:21.097242 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:21.128613 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:21.133399 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:21.141398 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:21.147491 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:21.148118 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:21.167182 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:21.170713 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:21.178988 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:21.287139 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:21.292100 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:21.310547 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:21.315337 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:21.318639 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:21.332046 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:21.332923 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:01:21.333654 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:21.336879 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:21.361421 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:21.367967 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:21.378376 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:21.384006 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:21.395309 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:21.400995 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:21.458982 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:21.459439 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:21.466064 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:21.480528 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:21.482064 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:21.488916 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:21.561072 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:21.565212 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:21.568926 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:21.572881 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:21.578184 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:21.578541 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:21.579907 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:21.581468 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:21.588178 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:21.593676 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:21.614784 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:21.622037 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:21.622758 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:21.636058 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:21.640585 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:21.699021 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:21.746116 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:21.747228 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:21.752862 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:21.769105 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:21.771030 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:21.772161 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:21.799355 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:21.811662 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:21.813845 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:21.841648 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:21.844736 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:21.845032 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:21.847223 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:21.861932 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:21.872073 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:21.876912 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 83 similar messages]
I20240501 08:01:21.877663 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:21.933766 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 08:01:21.935926 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:21.939074 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:21.944703 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:21.947086 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:21.954960 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:21.966696 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:21.972440 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:22.019971 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:22.019989 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:22.020427 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:22.026175 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:22.031044 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:22.035039 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:22.047349 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:22.048944 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:22.115146 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:22.118899 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:22.122627 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:22.124559 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:22.136847 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:22.140383 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:22.212847 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:22.219401 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:22.221881 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:22.228389 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:22.229163 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:22.253603 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:22.274788 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:22.301857 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:22.302708 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:22.307493 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:22.312585 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:22.319897 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:22.328538 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:22.330080 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:22.373927 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:22.377207 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:22.387707 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:22.389536 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:22.392791 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:22.393234 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:22.399189 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:22.400017 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:22.452275 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:22.469051 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:22.470762 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:22.471908 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:22.472316 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:22.472358 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:22.474916 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:22.481027 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:22.555428 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:01:22.574214 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:22.575719 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:22.582387 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:22.597028 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:22.602161 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:22.612687 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:22.618275 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:01:22.618629 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:22.625588 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:22.634409 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:22.666281 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:22.669631 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:22.672896 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:22.686653 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:22.695806 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:22.703910 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:22.776607 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:22.779860 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:22.782398 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:22.790627 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:22.793371 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:22.796350 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:22.797782 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:22.798502 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:22.845566 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:22.872661 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:22.876685 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:22.885236 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:22.888533 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:22.889443 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:22.890549 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:22.898280 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:22.901945 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:22.960925 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:22.961454 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:22.966993 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:22.973471 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:22.982760 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:22.985818 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:23.052892 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:23.055262 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:23.055778 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:23.056465 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:23.058785 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:23.062912 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:23.076815 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:23.122536 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:23.153218 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:23.154090 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:23.157474 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:23.168292 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:23.179991 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:23.180999 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:23.199424 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 08:01:23.213353 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:23.220908 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:23.222954 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:23.225898 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:23.234948 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:23.238885 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:23.244540 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:23.298542 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:23.300355 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:23.319566 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:23.321410 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:23.324810 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:23.327952 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:23.334353 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:23.338879 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:23.403676 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:23.405633 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:23.407086 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:23.407495 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:23.409847 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:23.419704 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:23.428104 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:23.526412 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:23.526871 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:23.527292 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:23.536845 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:23.537186 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:23.540144 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:23.561295 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:23.607714 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:23.610549 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:23.611238 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:23.611392 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:23.612948 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:23.623876 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:23.631558 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:01:23.636253 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:23.638214 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:23.672287 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:23.717684 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:23.730698 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:23.733003 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:23.743589 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:23.751660 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:23.765583 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:23.772606 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:23.777340 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:23.787272 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:23.788805 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:23.790606 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:23.797291 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:01:23.808432 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:23.820626 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:23.932476 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:23.933734 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:23.938593 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:23.944876 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:23.948684 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:23.955785 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:23.956183 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:23.956233 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:23.959667 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:23.974766 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:23.983392 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:23.991397 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:24.052917 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:24.072829 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:24.072829 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:24.074615 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:24.092473 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:24.093312 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:24.101626 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:24.119007 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:24.138918 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:24.186204 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:24.188141 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:24.193063 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:24.196525 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:24.203325 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:24.207791 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:24.254526 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:24.254743 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:24.256181 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:24.257027 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:24.260414 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:24.261081 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:24.278513 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:24.292245 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:24.371719 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:01:24.375782 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:24.384682 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:24.388656 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:24.393679 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:24.400177 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:24.410722 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:24.431592 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:24.445931 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:01:24.451493 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:24.453366 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:24.465039 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:24.468346 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:24.491071 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:24.516419 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:24.519922 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:24.525270 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:24.527870 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:24.528267 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:24.534600 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:24.536235 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:24.555393 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:24.628717 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:24.629345 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:24.629727 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:24.633877 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:24.636263 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:24.643358 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:24.646538 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:24.755815 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:24.763872 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:24.773252 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:24.776932 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:24.780716 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:24.800494 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:01:24.807798 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:24.814627 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:24.817101 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:24.822531 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:24.828735 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:24.831990 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:01:24.839736 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:24.840127 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:24.851054 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:24.927069 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:01:24.950817 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:24.951156 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:01:24.960505 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:24.966868 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:24.980933 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:01:24.989578 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:24.992658 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:25.014629 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:25.021493 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:25.030086 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:25.037973 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:25.049644 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:25.053541 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:25.054185 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:25.059209 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:25.082274 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:25.083546 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:25.085242 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:25.090132 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:25.115072 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:25.123742 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:25.135526 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:25.147007 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:25.148370 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:25.163007 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:25.164566 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:25.164662 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:25.201007 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:25.215358 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:25.216260 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:25.216949 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:25.226200 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:25.226612 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:25.234663 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:25.297622 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:25.305552 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:25.337921 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:25.345165 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:25.345697 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:01:25.347424 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:25.350044 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:25.369016 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:25.370114 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:25.446134 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:25.459903 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:25.460604 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:25.460968 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:25.469177 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:25.469624 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:25.473353 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:25.527980 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:01:25.534088 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:25.534588 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:25.545954 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:25.548265 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:25.559839 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:25.564440 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:25.608065 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:25.610816 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:25.619750 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:25.621323 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:25.626500 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:25.634084 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:25.714205 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:25.715245 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:25.717155 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:25.717862 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:25.719319 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:25.742391 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:25.763671 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:01:25.815721 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:25.816026 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:25.827428 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:25.843381 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:25.854357 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:25.866483 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:25.880185 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:01:25.894975 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:25.895483 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:25.901103 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:25.907618 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:25.907776 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:25.911084 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:25.929020 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:25.971256 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:25.985987 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:25.988337 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:25.997331 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:26.000870 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:26.010636 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:26.015950 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:26.062312 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:26.069751 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:26.075143 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:26.075143 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:26.085613 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:26.094681 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 76 similar messages]
I20240501 08:01:26.106006 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:26.156736 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:01:26.178392 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:26.180598 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:26.180604 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:26.191514 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:26.193647 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:26.195174 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:26.219066 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:01:26.220966 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:26.281867 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:26.284245 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:26.284775 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:26.286077 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:26.291762 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:26.302042 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:01:26.325847 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:26.333117 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:26.350275 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:01:26.379050 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:26.385293 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:26.386852 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:26.391463 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:26.396915 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:26.406832 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:26.411866 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:26.412494 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:26.417835 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:26.421288 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:26.439301 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:01:26.440603 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:26.441954 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:01:26.455541 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:26.546052 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:26.549036 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:26.552457 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:26.555889 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:26.556684 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:26.559593 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:26.566807 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:26.662482 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:26.665638 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:26.668002 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:26.675045 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:26.679509 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:26.684674 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:26.704578 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:26.738880 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:01:26.747912 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:26.752527 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:26.759821 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:26.762385 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:26.766018 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:26.788583 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:26.808822 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:26.857028 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:01:26.881595 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:26.883069 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:26.885752 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:26.892350 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:26.892580 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:26.892867 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:26.902379 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:26.904057 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:26.906282 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:26.912328 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:26.915066 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:26.924942 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:01:26.926003 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:26.965447 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:01:26.966236 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:26.966682 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:26.983366 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:27.056143 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:27.056449 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:27.056449 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240501 08:01:27.061146 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:27.063171 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:27.064342 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:27.066581 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:27.154289 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:27.158401 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:27.167840 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:27.168550 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:27.172353 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:27.189929 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:27.223860 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:27.224864 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:27.227382 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:27.236471 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:27.237026 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:27.240263 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:27.246436 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:27.256731 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:27.343034 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:27.353821 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:27.354642 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:27.358038 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:27.378923 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:27.382972 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:27.396296 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:27.398561 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:27.406023 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:27.413070 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:01:27.416477 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:27.420923 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:27.431399 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:27.450750 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:27.462299 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:27.468271 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:27.481154 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:27.484623 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:27.484683 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:27.488544 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:01:27.496268 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:27.498613 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:27.510898 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:27.526958 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:27.554392 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:27.563603 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:27.608350 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:27.616714 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:27.616917 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:27.633426 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:27.635685 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:27.636072 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:27.658095 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:27.662035 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:27.702709 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:27.704023 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:27.709067 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:27.714179 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:27.720014 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:27.731801 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:27.777652 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:27.797963 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:27.799613 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:27.805737 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:27.806150 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:27.808748 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:27.819131 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:27.827754 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:27.829411 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:27.842628 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:27.874001 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:27.913502 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:27.928992 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:27.940026 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:27.940697 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:27.948591 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:27.955876 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:27.956748 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:27.967605 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:27.972231 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:27.980242 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:27.981339 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:28.006592 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:28.040776 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:28.041059 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:28.061131 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:28.063808 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:28.065500 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:28.067840 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:28.069967 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:28.117094 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:28.172237 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:28.176404 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:28.177554 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:28.178481 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:28.198637 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:28.213057 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:28.217043 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:28.220221 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:28.220502 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:28.221793 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:28.243271 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:28.253672 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:28.261307 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:28.266155 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:28.295737 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:01:28.301695 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:28.359370 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:28.360873 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:28.367503 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:28.369629 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:28.376952 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:28.390172 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:28.431344 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:28.435137 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:28.438622 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:28.445237 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:28.446538 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:28.450611 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:28.451263 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:28.516005 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:28.583271 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:28.584686 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:28.584985 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:28.589738 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:28.596940 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:28.609463 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:28.688284 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:28.744446 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:28.750090 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:28.752857 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:28.753039 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:28.762708 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:01:28.774526 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:28.777190 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:28.781853 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:28.782317 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:28.794262 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:28.799680 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:28.805400 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:28.805714 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:28.806277 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:28.838891 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:28.902829 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:28.903064 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:28.906662 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:28.912004 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:28.924964 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:28.934629 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:28.966243 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:28.992089 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:28.993016 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:28.994882 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:29.001435 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:29.016806 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:29.021332 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:29.023885 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:29.045866 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:29.082149 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:29.085305 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:29.093788 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:29.096993 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:29.099277 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:29.102689 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:29.114323 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:29.123109 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:29.195443 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:29.205261 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:29.205461 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:29.220796 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:29.224107 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:29.229697 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:29.239292 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:29.261248 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:29.266168 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:29.267390 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:29.270145 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:29.280680 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:29.304600 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:01:29.319689 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:29.331843 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:29.333962 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:29.335847 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:29.338665 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:29.347353 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:29.349880 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:29.354005 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:29.372845 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:29.436568 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:29.488461 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:29.489584 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:29.496361 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:29.497124 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:29.505333 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:29.512383 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:29.520218 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:29.550248 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:29.560731 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:29.563449 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:29.564080 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:29.565352 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:29.580360 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:29.640312 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:29.649956 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:29.677640 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:29.694525 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:29.701961 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:29.702085 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:29.708266 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:29.726500 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:29.735095 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:29.761232 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:29.761222 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:29.775847 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:29.780583 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:29.783735 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:29.784524 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:29.785092 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:29.808342 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:29.840857 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:29.842556 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:29.847337 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:29.848044 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:29.876188 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:29.879029 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:29.881220 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:29.882864 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:29.886953 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:29.968442 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:29.969509 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:29.970288 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:29.972725 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:29.979946 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:29.982612 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:30.013902 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:01:30.014806 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:30.017541 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:30.019296 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:30.020284 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:30.026616 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:30.027896 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:30.055548 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:30.116147 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:30.120222 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:30.148906 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:30.175207 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:30.176748 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:30.177299 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:30.180600 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:30.187707 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:30.197188 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:30.211557 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:30.213213 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:30.217096 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:30.230865 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:30.231561 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:30.238101 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:30.238479 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:30.240406 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:30.292085 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:30.312101 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:30.315349 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:30.319375 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:30.324838 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:30.325217 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:30.335422 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:30.523622 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:30.526958 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:30.547559 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:30.563684 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:30.566176 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:30.575475 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:30.575629 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:30.583145 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:30.591732 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:30.593740 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:30.665607 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:30.668462 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:30.670962 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:30.674747 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:30.675381 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:30.675732 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:30.702881 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:30.725173 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:30.741007 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:30.749200 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:30.751032 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:30.753181 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:30.765046 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:30.835032 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:30.835050 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:30.839630 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:30.856487 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:30.857262 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:30.881688 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:30.900838 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:30.915838 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:30.919845 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:30.920405 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:30.928979 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:30.929246 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:30.936673 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:30.937893 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:30.940830 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:31.033533 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:31.037683 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:31.039011 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:31.051115 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:31.054502 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:31.058954 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:31.079202 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:31.114578 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:31.119417 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:31.135080 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:31.140182 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:31.142385 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:31.147544 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:31.161782 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:31.188799 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 83 similar messages]
I20240501 08:01:31.198729 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:31.201691 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:31.204767 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:31.214663 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:31.217386 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:31.254241 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:31.295470 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:31.304083 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:31.304446 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:31.309576 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:31.322661 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:31.323988 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:31.335245 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:31.355571 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:31.362329 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:31.365890 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:31.366647 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:31.368088 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:31.389851 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:31.407372 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:31.414615 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:01:31.415311 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:01:31.457882 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:31.505200 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:31.505595 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:31.506655 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:31.511723 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:31.525866 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:31.529819 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:31.531841 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:31.543905 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:31.546280 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:31.553500 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:31.565651 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:31.579325 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:31.589205 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:31.610967 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:31.619506 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:31.666858 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:31.668864 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:31.672384 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:31.677341 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:31.679246 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:31.687131 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:31.696728 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:31.697926 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:31.743667 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:31.749243 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:31.779395 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:31.788945 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:31.792588 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:31.792851 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:31.796705 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:31.798099 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:31.841348 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:31.849330 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:31.852952 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:31.858639 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:31.862390 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:31.862557 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:31.869459 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:31.878770 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:31.910207 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:31.977953 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:31.991205 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:31.993736 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:31.999639 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:32.000531 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:32.019155 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:32.023495 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:32.048862 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:32.050595 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:32.054702 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:32.063167 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:32.078769 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:32.115276 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 08:01:32.119278 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:32.121294 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:32.121901 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:32.131215 11598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 08:01:32.135360 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:32.148619 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:32.155440 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:32.169855 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:32.228685 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:32.230604 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:32.230896 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:32.236639 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:32.236907 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:32.238863 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:32.308625 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:32.325913 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:32.330056 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:32.348798 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:32.349018 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:32.349274 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:32.351380 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:32.353948 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:32.375677 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:32.415423 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:32.471411 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:32.473771 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:32.474591 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:32.476758 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:32.497538 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:32.503388 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:32.503769 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:32.521116 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:32.523123 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:32.523653 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:32.531675 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:32.535343 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:32.546581 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:32.568717 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:32.647661 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:32.647984 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:32.648041 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:32.655887 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:32.661762 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:32.667793 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:32.667999 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:32.684078 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:01:32.698350 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:32.707747 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:32.713217 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:32.722640 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:32.732079 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:32.737733 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:32.747514 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:32.748076 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:32.767976 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:32.868808 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:32.869889 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:32.871595 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:32.877873 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:32.878602 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:32.897044 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:32.924935 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:32.934671 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:32.955603 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:32.957931 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:32.958761 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:32.960815 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:32.965783 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:32.982818 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:32.985967 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:32.989441 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:33.000921 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:33.001024 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:33.001497 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:33.019590 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:33.030385 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:33.035310 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:33.069269 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:33.106747 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:33.107585 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:33.107610 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:33.116228 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:33.130621 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:33.140046 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:33.189448 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:33.193415 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:33.196692 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:33.205302 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:33.229086 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:33.229429 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:33.263482 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:33.297508 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:33.299302 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:33.303835 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:33.305791 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:33.307232 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:33.312405 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:33.344414 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:33.388150 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:33.407600 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:01:33.418315 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:33.425035 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:33.426193 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:33.433101 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:33.433696 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:33.438061 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:33.439690 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:33.442286 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:33.448890 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:33.460099 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:33.462165 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:33.476094 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:33.476410 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:33.547560 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:33.550117 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:33.557531 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:33.561677 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:33.561825 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:33.564332 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:33.572597 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:33.601150 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:33.662771 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:33.666512 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:33.669740 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:33.677976 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:33.688957 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:33.716600 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:33.721634 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:33.755261 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:33.758133 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:33.759654 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:33.762987 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:33.763819 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:33.785385 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:33.793509 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:33.858873 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:33.860407 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:33.869341 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:33.870626 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:33.874702 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:33.882714 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:33.901930 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:33.938666 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:33.940297 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:33.944099 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:33.944178 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:33.947278 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:33.950973 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:01:33.959012 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:33.979035 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:34.032698 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:34.043438 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:01:34.059578 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:34.064458 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:34.068347 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:34.068423 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:34.082558 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:34.094759 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:34.096761 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:34.103940 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:34.114645 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:34.123243 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:34.126895 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:34.140786 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:34.141463 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:34.145784 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:34.181463 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:34.228318 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:34.232970 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:34.236863 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:34.237741 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:34.239244 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:34.242439 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:34.244246 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:34.285009 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:34.300011 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:34.300794 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:34.301558 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:34.301827 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:34.302839 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:34.313385 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:34.316540 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:34.337333 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:34.345187 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:34.361411 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:01:34.404901 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:01:34.431752 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:34.432984 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:34.447134 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:34.453012 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:34.454154 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:34.457901 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:34.503420 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:34.512701 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:34.515518 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:34.516839 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:34.535815 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:34.540045 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:34.565336 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:34.572005 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:34.617061 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:01:34.649293 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:34.651799 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:34.652294 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:34.662951 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:34.674003 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:34.678555 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:34.702587 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:34.705067 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:34.708173 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:34.717892 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:34.735016 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:34.752004 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:34.786952 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:34.789088 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:34.791956 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:34.804383 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:34.809641 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:34.810724 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:34.819837 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:34.823808 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:34.909916 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:34.911778 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:34.921859 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:34.928645 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:34.934727 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:34.937585 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:34.960462 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:34.960928 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:34.963582 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:34.977640 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:34.999527 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:01:34.999933 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:35.007128 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:35.008361 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:35.091670 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:35.103507 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:35.107326 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:35.108594 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:35.125789 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:01:35.143025 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:35.143673 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:35.151206 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:35.164021 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:01:35.164342 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:35.176839 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:35.184911 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:01:35.191807 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:35.193781 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:35.200551 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:01:35.202993 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:35.204525 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:35.206568 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:35.209543 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:35.225077 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:35.230026 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:35.250867 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:35.274325 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:35.286541 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:35.373603 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:35.376077 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:35.384711 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:35.386826 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:35.389541 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:35.391204 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:35.401654 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:35.406651 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:35.415052 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:35.419289 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:35.431110 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:35.439307 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:35.454021 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:35.484263 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:01:35.565001 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:35.567010 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:35.574939 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:35.576073 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:35.576427 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:35.581624 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:35.581732 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:35.588963 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:35.600122 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:35.611244 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:35.613184 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:35.621356 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:35.648751 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:35.683555 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:35.716941 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:01:35.751026 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:35.758998 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:35.759680 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:35.766670 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:35.770846 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:35.786329 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:35.800611 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:35.804687 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:35.805795 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:35.812353 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:35.820444 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:35.830308 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:35.897562 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:35.907315 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:35.955401 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:01:35.974320 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:35.984380 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:35.990348 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:35.990624 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:36.002943 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:36.006795 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:36.009244 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:36.030239 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:36.034134 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:36.039532 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:36.041025 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:36.050202 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:36.050935 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:36.051440 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:36.060921 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:36.065985 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:36.067595 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:36.068353 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:36.117483 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:36.117595 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:36.147285 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:01:36.202208 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:36.202718 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:36.204257 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:36.206863 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:36.212000 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:36.226289 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240501 08:01:36.228266 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:36.243808 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:36.274677 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:36.327024 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:36.329788 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:36.332274 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:36.332267 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:36.340677 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:36.350844 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:36.362551 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:36.384645 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:36.452620 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:36.455214 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:36.471050 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:36.479183 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:36.484164 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:36.488726 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:36.489145 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:36.491102 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:36.491437 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:36.495169 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:01:36.503703 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:36.539232 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:36.548867 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:36.563863 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:36.592741 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:01:36.611395 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:36.617393 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:36.617641 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:36.619493 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:36.620592 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:36.639032 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:36.647362 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:01:36.695875 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:36.696396 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:36.701756 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:36.702178 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:36.716198 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:36.716482 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:36.719846 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:01:36.721869 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:36.741649 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:36.808391 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:36.809860 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:36.812486 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:36.813033 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:36.822297 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:36.823168 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:36.871985 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:01:36.916558 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:36.929817 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:36.929895 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:36.939106 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:36.939587 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:36.941658 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:36.946273 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:36.958074 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:36.980393 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:36.983312 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:36.985811 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:36.989750 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:36.995163 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:37.014189 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:37.038208 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:37.068663 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:37.068912 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:37.073881 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:37.080013 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:37.082139 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:37.082943 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:01:37.088164 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:37.091025 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:37.098696 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:37.116408 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 08:01:37.148954 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:37.153213 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:37.153626 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:37.157917 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:37.162613 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:37.163224 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:37.177889 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 08:01:37.202801 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:37.283931 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:37.284797 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:37.291679 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:37.292663 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:37.308856 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:01:37.311012 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:37.320894 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:37.340134 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:37.340337 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:01:37.388069 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:37.388096 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:37.390051 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:37.391391 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:37.398588 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:37.415936 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:37.468056 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:37.484946 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:37.487679 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:37.490885 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:37.506608 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:37.521185 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:37.528003 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:37.551096 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:37.555111 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:37.556349 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:37.565479 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:37.567255 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:37.580139 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:37.581648 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:37.592465 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:37.702286 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:37.705602 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:37.707505 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:37.712517 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:37.719792 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:37.728235 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:37.730370 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:37.736080 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:37.738991 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:37.753496 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:37.755965 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:37.773023 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:37.781811 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:37.815191 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:37.867971 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:37.880426 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:37.882290 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:37.888063 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:37.895253 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:37.896485 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:37.919281 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:37.922147 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:37.936451 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:37.972208 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:37.972579 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:37.980777 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:37.981431 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:37.986624 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:38.010457 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:38.011978 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:38.019232 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:38.024992 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:38.025084 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:38.029172 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:38.029397 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:38.043509 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:38.097430 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:38.160852 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:38.167834 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:38.168596 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:38.172734 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:38.182256 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:38.186602 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:38.195343 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:38.265153 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:38.272589 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:38.275583 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:38.283353 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:38.286715 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:38.292316 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:38.381043 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:38.383857 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:38.389906 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:38.390857 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:38.391626 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:38.415625 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:38.456903 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:38.459662 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:38.466192 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:38.472033 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:38.475422 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:38.479694 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:38.515447 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:38.521435 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:38.526043 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:01:38.527204 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:38.527441 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:38.573311 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:38.577737 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:38.580870 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:38.585081 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:38.595562 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:38.606053 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:38.617808 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:38.682644 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:38.683493 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:38.684469 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:38.690232 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:38.695621 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:38.696564 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:38.721415 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:38.726585 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:38.742985 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:38.760460 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:38.762668 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:38.764847 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:38.779726 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:38.783169 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:38.799079 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:01:38.833580 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:38.838501 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:38.839361 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:38.841940 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:38.844534 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:38.846756 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:38.940608 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:38.940604 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:38.945878 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:38.960213 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:38.963526 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:38.967206 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:39.008850 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:39.032459 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:39.035198 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:39.068521 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:39.073634 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:39.076129 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:39.078879 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:39.080050 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:39.099639 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:39.101747 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:39.106041 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:39.139801 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:39.145735 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:39.146804 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:39.155864 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:39.167538 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:39.171464 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:39.184644 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:39.185308 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:39.190296 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:39.250411 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:39.253067 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:39.257712 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:39.265431 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:39.266363 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:39.272658 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:39.273548 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:01:39.287331 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:39.333937 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:01:39.338871 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:39.348290 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:39.351863 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:39.353977 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:39.362365 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:39.384753 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:39.388666 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:39.422233 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 08:01:39.428743 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:39.434314 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:39.440883 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:39.445758 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:39.446103 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:39.449461 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:39.468639 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:39.532828 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:39.579533 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:39.601294 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:39.618582 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:39.623750 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:39.630110 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:39.646353 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:39.647954 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:39.650103 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:39.650651 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:39.652117 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:39.666764 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:39.683724 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:39.686029 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:01:39.694701 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:39.703295 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:39.710956 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:39.712982 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:39.715384 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:39.723608 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:39.743033 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:39.764575 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:39.841293 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:01:39.889108 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:39.889475 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:39.893342 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:39.893839 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:39.898592 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:39.899463 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:39.901469 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:39.911221 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:39.913751 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:39.941701 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:39.941957 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:39.954664 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:40.024746 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:01:40.074254 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:40.076342 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:40.081358 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:40.082599 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:40.086458 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:40.089428 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:40.110507 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:40.122504 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:40.147495 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:01:40.148941 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:40.153383 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:40.156189 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:40.156507 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:40.156508 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:40.166985 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:40.171942 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:40.175676 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:40.189755 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:40.194255 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:40.197484 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:40.203193 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:40.242815 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:40.245029 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:40.245597 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:01:40.327004 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:40.330034 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:40.330205 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:40.342814 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:01:40.343716 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:40.345602 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:40.346653 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:40.471714 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:40.473083 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:40.480994 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:40.481451 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:40.491367 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:01:40.503173 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:40.505517 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:40.517779 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:40.519161 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:40.522794 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:40.530886 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:40.546594 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:01:40.553218 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:40.571478 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:40.575289 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:40.589754 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:40.615598 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:40.619369 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:40.621695 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:40.624874 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:01:40.629942 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:40.638953 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:40.647688 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:40.723423 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:40.728787 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:40.729647 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:40.730953 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:40.738646 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:40.741452 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:40.863415 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:40.865857 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:01:40.867965 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:40.870643 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:40.875089 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:40.879532 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:40.883827 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:40.959280 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:40.964581 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:40.967672 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:40.972465 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:40.975339 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:40.979768 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:40.983933 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:41.000181 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:41.033452 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:01:41.045035 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:41.057516 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:41.057842 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:41.063709 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:41.073410 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:41.073984 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:41.089675 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:41.097136 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:01:41.114217 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:41.118467 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:41.122318 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:41.129051 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:41.130291 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:41.143520 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:41.204048 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:41.206245 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:41.209750 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:41.219141 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:41.219796 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:01:41.219915 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:41.224691 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:41.227341 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:41.227411 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:01:41.240151 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:01:41.241056 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:41.305994 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:41.310883 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:41.316505 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:41.319998 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:41.325798 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:41.328457 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:41.388765 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:41.437894 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:41.438284 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:41.442836 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:41.460660 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:41.465135 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:41.481314 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:41.486193 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:41.487643 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:41.487926 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:41.490782 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:41.491544 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:41.494658 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:41.503918 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:41.514359 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:41.528589 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:01:41.588009 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:01:41.601020 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:41.603482 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:41.607291 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:41.609292 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:41.610101 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:41.639222 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:41.676620 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:41.681568 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:01:41.690336 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:41.701524 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:01:41.705650 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:41.706619 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:41.707052 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:41.714668 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:41.721966 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:01:41.730834 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:41.749804 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:41.782521 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:41.787504 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:41.809725 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:41.849478 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:01:41.854854 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:41.859074 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:41.861824 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:41.877494 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:41.879477 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:41.935537 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:41.953109 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:41.955005 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:41.955855 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:41.958393 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:41.960090 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:41.981868 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:41.985716 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:42.032116 11117 maintenance_manager.cc:419] P 2ed840e35a9f4afe9077d15510856d4e: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033397
I20240501 08:01:42.043656 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:42.045914 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:42.048772 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:42.059592 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:42.062016 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:42.065104 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:01:42.075064 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:42.093681 11116 maintenance_manager.cc:643] P 2ed840e35a9f4afe9077d15510856d4e: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.061s user 0.026s sys 0.000s Metrics: {"bytes_written":20643,"cfile_init":1,"dirs.queue_time_us":8762,"dirs.run_cpu_time_us":277,"dirs.run_wall_time_us":1866,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":907,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":43,"thread_start_us":202,"threads_started":2,"wal-append.queue_time_us":1387}
I20240501 08:01:42.095753 11117 maintenance_manager.cc:419] P 2ed840e35a9f4afe9077d15510856d4e: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7332 bytes on disk
I20240501 08:01:42.096995 11116 maintenance_manager.cc:643] P 2ed840e35a9f4afe9077d15510856d4e: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":177,"lbm_reads_lt_1ms":4}
I20240501 08:01:42.179241 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:42.179319 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:42.182507 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:42.187011 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:42.187463 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:42.197108 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 08:01:42.206966 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:42.212363 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:42.217144 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:42.222725 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:42.240654 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:42.246716 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:01:42.253376 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:42.257894 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:42.350510 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:01:42.351280 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:42.359088 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:42.361506 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:42.364475 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:42.377243 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:42.380082 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:42.388164 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:42.430317 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:42.434069 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:42.441308 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:42.443233 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:42.444319 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:42.447055 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:42.480186 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:42.551699 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:42.562337 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:42.563133 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:42.563961 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:42.564882 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:42.577348 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:42.598717 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:42.621984 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:01:42.708364 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:01:42.720935 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:42.723309 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:42.737555 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:42.760880 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:42.762894 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:01:42.769932 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:42.771557 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:42.778272 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:42.778916 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:42.780824 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:42.790829 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:42.792688 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:42.795241 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:01:42.809060 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:42.823990 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:42.884963 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:42.889863 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:42.893292 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:42.898308 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:42.901942 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:42.927016 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:42.930248 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:42.963825 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:42.968466 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:01:42.984680 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:42.993621 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:42.994972 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:42.995460 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:42.995486 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:42.996766 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:43.050824 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:43.056051 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:43.067359 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:43.067781 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:43.069280 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:43.071238 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:43.090323 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:43.119012 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:43.208904 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:43.212373 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:43.214134 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:43.221765 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:43.232851 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:43.249749 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:43.253917 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:43.254778 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:43.256655 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:43.258625 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:43.270123 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:43.278877 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:43.283660 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:43.292013 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:43.303010 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:43.332587 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:43.333871 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:43.344292 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:43.362989 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:43.461661 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:43.471452 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:43.473239 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:43.477878 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:43.479692 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:43.490198 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:43.497977 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:43.504109 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:43.506026 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:43.509747 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:43.515970 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:01:43.525609 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:43.530575 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:43.534826 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:43.538902 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:43.613981 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:43.642026 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:01:43.642832 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:43.679097 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:01:43.688661 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:43.691342 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:43.698110 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:43.719326 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:43.729447 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:43.735736 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:43.750473 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:43.756716 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:43.766060 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:43.766057 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:43.767912 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:43.778667 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:43.779274 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:43.782320 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:43.785473 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:43.794821 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:43.807636 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:01:43.809917 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:43.843397 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:43.845889 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:43.855221 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:43.934823 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:01:43.949808 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:43.959983 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:43.968020 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:43.974983 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:43.986333 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:43.988070 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:43.988085 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:43.999068 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:43.999483 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:44.007133 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:44.018179 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:44.037621 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:44.069579 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:44.070183 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:44.085464 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:44.093602 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:44.146407 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:44.157456 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:01:44.163630 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:44.164822 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:01:44.186967 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:44.190739 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:44.198479 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:44.209424 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:44.214195 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:44.221436 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:44.224470 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:44.231230 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:44.239830 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:44.241350 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:44.243485 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:44.251084 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:44.256557 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:44.257508 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:44.257920 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:44.276414 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:01:44.277101 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:44.296840 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:44.298247 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:44.299105 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:44.335436 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:01:44.353854 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:44.419751 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:44.618247 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:44.652626 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:44.653457 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:44.653993 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:44.656270 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:44.669652 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:44.672432 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:01:44.680864 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:44.686843 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:44.698477 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:44.702049 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:44.720506 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:44.724033 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:44.728092 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:44.731325 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:44.740221 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:44.741943 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:44.743242 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:44.753058 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:44.754839 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:44.790216 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:44.825445 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:44.886934 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:44.914690 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:44.917119 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:44.929739 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:44.933121 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:44.941881 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:44.943490 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:44.978229 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:44.979990 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:44.988194 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:44.988940 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:44.994283 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:45.004649 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:45.110733 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:45.112406 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:45.130992 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:45.131947 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:45.134828 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:45.138334 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:45.142967 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:45.148566 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:45.151675 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:45.155054 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:45.164541 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:45.191610 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:45.198171 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:45.211649 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:45.222771 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:45.225461 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:45.229624 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:45.247918 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:45.248224 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:45.259150 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:45.261888 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:45.267938 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:45.316224 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:45.363039 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:45.370179 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:45.370507 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:45.370609 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:45.380676 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:45.384528 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:45.408876 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:01:45.434351 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:45.436110 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:45.438068 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:45.438647 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:01:45.442633 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:45.443890 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:45.463359 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:45.484189 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:45.535352 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:45.543591 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:45.552439 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:45.562191 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:45.564221 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:45.564527 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:45.623778 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:45.633437 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:45.636009 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:45.641597 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:45.658020 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:45.658098 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:45.660197 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:45.678545 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:45.681200 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:45.692720 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:45.699990 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:45.701526 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:45.713375 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:45.727361 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:45.730731 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:45.730988 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:45.842051 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:45.844280 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:45.846544 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:45.857978 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:45.862234 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:45.867393 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:45.874598 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:45.902159 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:45.902880 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:45.907084 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:45.913662 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:45.920217 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:45.933952 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:45.955440 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:45.970146 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:46.027807 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:46.036823 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:46.040868 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:46.059257 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:46.064824 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:46.066017 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:46.079186 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:46.097167 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:46.101135 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:46.109397 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:46.119453 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:46.123107 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:46.128815 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:01:46.134402 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:46.143805 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:46.144719 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:46.173902 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:46.173964 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:46.177016 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:46.180684 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:46.185688 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:46.198614 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:46.199667 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:46.275286 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 08:01:46.303682 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:46.305148 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:46.307440 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:46.308609 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:46.316726 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:46.348559 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:46.352936 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:01:46.423524 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:46.425406 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:46.426295 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:46.427475 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:46.434595 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:46.438336 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:46.439848 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:46.444911 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:46.525621 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:46.527861 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:46.528857 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:46.529764 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:46.533437 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:46.554574 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:46.562491 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:01:46.563015 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:01:46.570492 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:46.615090 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:46.615854 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:46.623765 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:46.624687 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:46.630288 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:46.630368 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:46.646627 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:46.647586 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:46.701821 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:46.709609 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:46.717141 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:46.721050 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:46.724640 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:46.726675 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:46.727805 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:46.737967 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:01:46.770269 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:01:46.773229 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:01:46.808691 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:46.823977 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:46.825111 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:46.827039 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:46.835755 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:46.836344 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:46.846320 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:01:46.855033 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:46.920100 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:46.920792 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:46.923213 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:46.924870 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:46.933377 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:46.951402 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:46.956857 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:46.977401 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:46.983219 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:46.986873 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:46.988948 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:46.993906 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:47.008564 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:47.018445 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:47.052484 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:47.055648 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:47.059980 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:47.060664 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:47.079799 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:47.086124 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:01:47.093452 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:47.111176 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:01:47.143574 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:47.172556 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:47.177217 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:47.179250 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:47.181152 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:47.194625 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:47.199255 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:47.233011 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240501 08:01:47.256318 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:47.259148 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:47.264142 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:47.265657 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling FlushMRSOp(c95c4f3ef08f44d1b69fb7b98426f5af): perf score=0.033334
I20240501 08:01:47.278155 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:47.281817 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling FlushMRSOp(a1ebf29682024645978b3caa2d4988f3): perf score=0.033341
I20240501 08:01:47.283587 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:47.305224 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:47.375680 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling FlushMRSOp(a24be92aec7544089c7818f2802b9c85): perf score=0.033366
W20240501 08:01:47.383021 11598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 08:01:47.389294 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: FlushMRSOp(c95c4f3ef08f44d1b69fb7b98426f5af) complete. Timing: real 0.123s user 0.030s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":7407,"dirs.queue_time_us":70486,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":630,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":254,"threads_started":3,"wal-append.queue_time_us":38370}
I20240501 08:01:47.390679 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling UndoDeltaBlockGCOp(c95c4f3ef08f44d1b69fb7b98426f5af): 4926 bytes on disk
I20240501 08:01:47.391364 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: UndoDeltaBlockGCOp(c95c4f3ef08f44d1b69fb7b98426f5af) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240501 08:01:47.419657 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:47.420186 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: FlushMRSOp(a1ebf29682024645978b3caa2d4988f3) complete. Timing: real 0.138s user 0.029s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":23373,"dirs.queue_time_us":35929,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":5951,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":746,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":541,"threads_started":3,"wal-append.queue_time_us":10674}
I20240501 08:01:47.421514 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling UndoDeltaBlockGCOp(a1ebf29682024645978b3caa2d4988f3): 4586 bytes on disk
I20240501 08:01:47.422150 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: UndoDeltaBlockGCOp(a1ebf29682024645978b3caa2d4988f3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240501 08:01:47.425781 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:47.431802 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:47.442598 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling FlushMRSOp(c95c4f3ef08f44d1b69fb7b98426f5af): perf score=0.033386
W20240501 08:01:47.445345 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:01:47.447280 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:47.448014 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:47.459901 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:47.517601 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:47.548805 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:47.596527 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:01:47.612854 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: FlushMRSOp(a24be92aec7544089c7818f2802b9c85) complete. Timing: real 0.237s user 0.035s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":19223,"dirs.queue_time_us":9504,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":1481,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":747,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":259,"threads_started":3,"wal-append.queue_time_us":9722}
I20240501 08:01:47.614094 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling UndoDeltaBlockGCOp(a24be92aec7544089c7818f2802b9c85): 4839 bytes on disk
I20240501 08:01:47.614800 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: UndoDeltaBlockGCOp(a24be92aec7544089c7818f2802b9c85) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240501 08:01:47.616361 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:47.634065 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:47.636173 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:01:47.636581 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:47.683791 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:47.687363 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:47.687433 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:47.696061 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:01:47.734947 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:01:47.735739 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: FlushMRSOp(c95c4f3ef08f44d1b69fb7b98426f5af) complete. Timing: real 0.293s user 0.035s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":72451,"dirs.queue_time_us":80951,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":1896,"drs_written":1,"lbm_read_time_us":409,"lbm_reads_lt_1ms":4,"lbm_write_time_us":603,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":341,"threads_started":3,"wal-append.queue_time_us":52308}
I20240501 08:01:47.737076 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling FlushMRSOp(a24be92aec7544089c7818f2802b9c85): perf score=0.033464
W20240501 08:01:47.761907 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:01:47.771785 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: FlushMRSOp(a24be92aec7544089c7818f2802b9c85) complete. Timing: real 0.034s user 0.020s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":1283,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":530,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":86,"threads_started":1,"wal-append.queue_time_us":177}
I20240501 08:01:47.773903 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling UndoDeltaBlockGCOp(c95c4f3ef08f44d1b69fb7b98426f5af): 4926 bytes on disk
I20240501 08:01:47.775395 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: UndoDeltaBlockGCOp(c95c4f3ef08f44d1b69fb7b98426f5af) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240501 08:01:47.776773 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling UndoDeltaBlockGCOp(a24be92aec7544089c7818f2802b9c85): 4839 bytes on disk
I20240501 08:01:47.777706 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: UndoDeltaBlockGCOp(a24be92aec7544089c7818f2802b9c85) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240501 08:01:47.816164 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:47.818262 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:47.818715 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:47.826081 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:47.831404 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:47.839953 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:47.865203 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:47.866199 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling FlushMRSOp(93602521de174997bda6702d25aafce1): perf score=0.033349
I20240501 08:01:47.891474 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: FlushMRSOp(93602521de174997bda6702d25aafce1) complete. Timing: real 0.025s user 0.021s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1231,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":581,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":79,"threads_started":1,"wal-append.queue_time_us":164}
I20240501 08:01:47.892463 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling FlushMRSOp(64613ea537c842f29d810e829278394e): perf score=0.033355
I20240501 08:01:47.893666 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling FlushMRSOp(b9431cf5853c4148ac38855b1c083d4f): perf score=0.033355
I20240501 08:01:47.910090 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:47.918157 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:01:47.919857 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:47.919996 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:47.920382 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:47.926072 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:47.927191 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling FlushMRSOp(64613ea537c842f29d810e829278394e): perf score=0.033363
I20240501 08:01:47.953512 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:47.979817 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: FlushMRSOp(64613ea537c842f29d810e829278394e) complete. Timing: real 0.052s user 0.016s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":20383,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":3043,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":650,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":190,"threads_started":2,"wal-append.queue_time_us":18234}
I20240501 08:01:47.981034 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling UndoDeltaBlockGCOp(64613ea537c842f29d810e829278394e): 4583 bytes on disk
I20240501 08:01:47.985219 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: UndoDeltaBlockGCOp(64613ea537c842f29d810e829278394e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":247,"lbm_reads_lt_1ms":4}
I20240501 08:01:47.995288 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: FlushMRSOp(b9431cf5853c4148ac38855b1c083d4f) complete. Timing: real 0.101s user 0.013s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":21120,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":2711,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":555,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":566,"threads_started":2,"wal-append.queue_time_us":14327}
I20240501 08:01:47.996333 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling UndoDeltaBlockGCOp(b9431cf5853c4148ac38855b1c083d4f): 4836 bytes on disk
I20240501 08:01:47.996862 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: UndoDeltaBlockGCOp(b9431cf5853c4148ac38855b1c083d4f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240501 08:01:47.999186 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: FlushMRSOp(64613ea537c842f29d810e829278394e) complete. Timing: real 0.106s user 0.019s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":11367,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":1157,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5565,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":118,"threads_started":1,"wal-append.queue_time_us":6724}
I20240501 08:01:48.000205 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling UndoDeltaBlockGCOp(93602521de174997bda6702d25aafce1): 4923 bytes on disk
I20240501 08:01:48.000662 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: UndoDeltaBlockGCOp(93602521de174997bda6702d25aafce1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240501 08:01:48.001391 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling UndoDeltaBlockGCOp(64613ea537c842f29d810e829278394e): 4583 bytes on disk
I20240501 08:01:48.001937 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: UndoDeltaBlockGCOp(64613ea537c842f29d810e829278394e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240501 08:01:48.031042 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling FlushMRSOp(93602521de174997bda6702d25aafce1): perf score=0.033394
I20240501 08:01:48.052354 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: FlushMRSOp(93602521de174997bda6702d25aafce1) complete. Timing: real 0.021s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":69,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":1469,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":448,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":113,"threads_started":1,"wal-append.queue_time_us":2217}
I20240501 08:01:48.053391 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling UndoDeltaBlockGCOp(93602521de174997bda6702d25aafce1): 4923 bytes on disk
I20240501 08:01:48.054327 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: UndoDeltaBlockGCOp(93602521de174997bda6702d25aafce1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240501 08:01:48.103118 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:48.104384 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:48.108871 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:48.113387 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:48.113916 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:48.114073 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:48.123055 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:01:48.132274 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:48.142701 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:48.145092 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:48.156401 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:48.157809 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:01:48.158141 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:48.168277 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:48.179704 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:48.237169 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling FlushMRSOp(f2e63d24d7df4fa3b2b26695e70ebe9f): perf score=0.033347
W20240501 08:01:48.243901 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:01:48.259969 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: FlushMRSOp(f2e63d24d7df4fa3b2b26695e70ebe9f) complete. Timing: real 0.023s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1790,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":670,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":112,"threads_started":1,"wal-append.queue_time_us":202}
I20240501 08:01:48.261066 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling UndoDeltaBlockGCOp(f2e63d24d7df4fa3b2b26695e70ebe9f): 4922 bytes on disk
I20240501 08:01:48.261710 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: UndoDeltaBlockGCOp(f2e63d24d7df4fa3b2b26695e70ebe9f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":209,"lbm_reads_lt_1ms":4}
I20240501 08:01:48.285845 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:48.293129 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:48.306041 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling FlushMRSOp(bd8348990f1b496bb62d8c0484193442): perf score=0.033369
I20240501 08:01:48.321028 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:48.324348 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:48.329459 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:48.335716 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:48.338651 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:48.340598 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:48.347424 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:48.354852 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:48.357543 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:48.363516 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:01:48.366693 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:48.390462 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: FlushMRSOp(bd8348990f1b496bb62d8c0484193442) complete. Timing: real 0.084s user 0.017s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2918,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1801,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":619,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":97,"threads_started":1,"wal-append.queue_time_us":40410}
I20240501 08:01:48.391636 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling UndoDeltaBlockGCOp(bd8348990f1b496bb62d8c0484193442): 4583 bytes on disk
I20240501 08:01:48.393427 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: UndoDeltaBlockGCOp(bd8348990f1b496bb62d8c0484193442) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240501 08:01:48.401026 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:48.504326 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:48.505088 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:48.507103 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:48.509663 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:48.521195 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:48.523370 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:48.540606 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:48.547374 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:48.549117 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:01:48.553185 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:48.556880 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:48.572659 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:48.575090 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:48.591051 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:01:48.602453 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:48.649098 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:01:48.653810 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240501 08:01:48.676151 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:48.678894 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:48.685652 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:48.691469 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:48.702605 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:48.706382 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:01:48.710937 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:48.752514 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:48.754557 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:48.756742 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:48.768116 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:48.790684 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:48.804625 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:01:48.809219 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:48.896306 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling FlushMRSOp(48b0ff0769dc426190f038811155c741): perf score=0.033348
I20240501 08:01:48.921770 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: FlushMRSOp(48b0ff0769dc426190f038811155c741) complete. Timing: real 0.025s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":8406,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":615,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":2300}
I20240501 08:01:48.922518 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:48.922675 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling UndoDeltaBlockGCOp(48b0ff0769dc426190f038811155c741): 4836 bytes on disk
I20240501 08:01:48.922909 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:48.927213 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: UndoDeltaBlockGCOp(48b0ff0769dc426190f038811155c741) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
W20240501 08:01:48.930513 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:48.932772 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:48.933557 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:48.945196 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:01:48.945501 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:01:48.953632 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:01:48.956962 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:48.958454 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:48.961247 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:48.963678 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:48.985306 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:48.990470 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:48.994236 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:48.994973 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:49.001034 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling FlushMRSOp(17bb1ff829bc47a6a0bd6030517652aa): perf score=0.033380
I20240501 08:01:49.005515 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling FlushMRSOp(867ba08a15b44d4c9e6fa92b0d518d1d): perf score=0.033382
I20240501 08:01:49.015012 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling FlushMRSOp(17bb1ff829bc47a6a0bd6030517652aa): perf score=0.033384
I20240501 08:01:49.044914 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: FlushMRSOp(17bb1ff829bc47a6a0bd6030517652aa) complete. Timing: real 0.030s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6062,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1389,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":619,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":163,"threads_started":2,"wal-append.queue_time_us":10071}
I20240501 08:01:49.046164 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling UndoDeltaBlockGCOp(17bb1ff829bc47a6a0bd6030517652aa): 4923 bytes on disk
I20240501 08:01:49.046733 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: UndoDeltaBlockGCOp(17bb1ff829bc47a6a0bd6030517652aa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
W20240501 08:01:49.047993 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:49.049414 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:49.053066 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: FlushMRSOp(867ba08a15b44d4c9e6fa92b0d518d1d) complete. Timing: real 0.047s user 0.010s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":8270,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1255,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":560,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":163,"threads_started":2,"wal-append.queue_time_us":8554}
I20240501 08:01:49.053066 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: FlushMRSOp(17bb1ff829bc47a6a0bd6030517652aa) complete. Timing: real 0.052s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":178,"dirs.run_cpu_time_us":136,"dirs.run_wall_time_us":1139,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":639,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":165,"threads_started":2,"wal-append.queue_time_us":8351}
I20240501 08:01:49.053954 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling UndoDeltaBlockGCOp(867ba08a15b44d4c9e6fa92b0d518d1d): 4583 bytes on disk
I20240501 08:01:49.054113 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling UndoDeltaBlockGCOp(17bb1ff829bc47a6a0bd6030517652aa): 4923 bytes on disk
I20240501 08:01:49.054652 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: UndoDeltaBlockGCOp(17bb1ff829bc47a6a0bd6030517652aa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240501 08:01:49.054721 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: UndoDeltaBlockGCOp(867ba08a15b44d4c9e6fa92b0d518d1d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240501 08:01:49.068574 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:01:49.151405 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:49.160708 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:49.175395 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:49.179767 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:49.189932 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:49.192736 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:01:49.192827 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:49.199640 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:49.199941 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:49.202800 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:49.231158 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:49.231451 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:49.235895 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:49.236557 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:49.238085 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:49.251614 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:49.258000 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:49.260804 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:49.272776 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:49.301553 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:01:49.354818 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:01:49.366760 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:49.414258 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:49.416114 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:01:49.424806 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:49.428390 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:49.430256 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:49.431782 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:49.439067 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:49.443714 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:49.447214 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:49.449234 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:49.451750 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:49.462122 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:49.477113 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:49.486608 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:49.498073 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:49.550870 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling FlushMRSOp(7a266f266c3d4fd49fa8b325f1045d61): perf score=0.033348
I20240501 08:01:49.556639 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling FlushMRSOp(7a266f266c3d4fd49fa8b325f1045d61): perf score=0.033350
W20240501 08:01:49.556897 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:49.556943 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling FlushMRSOp(f80db2e37bbc4716ad3a5200c61ee97b): perf score=0.033350
I20240501 08:01:49.571696 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: FlushMRSOp(7a266f266c3d4fd49fa8b325f1045d61) complete. Timing: real 0.021s user 0.016s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6206,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":582,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":163,"threads_started":2,"wal-append.queue_time_us":2435}
I20240501 08:01:49.573140 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling UndoDeltaBlockGCOp(7a266f266c3d4fd49fa8b325f1045d61): 4926 bytes on disk
I20240501 08:01:49.576421 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: UndoDeltaBlockGCOp(7a266f266c3d4fd49fa8b325f1045d61) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240501 08:01:49.577454 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling FlushMRSOp(b4707b952d174a9bb877ad8bd7a11268): perf score=0.033355
W20240501 08:01:49.589776 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:49.611223 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: FlushMRSOp(b4707b952d174a9bb877ad8bd7a11268) complete. Timing: real 0.034s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":436,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1289,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":576,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":61,"threads_started":1,"wal-append.queue_time_us":4838}
I20240501 08:01:49.614753 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling UndoDeltaBlockGCOp(b4707b952d174a9bb877ad8bd7a11268): 4586 bytes on disk
I20240501 08:01:49.615413 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: UndoDeltaBlockGCOp(b4707b952d174a9bb877ad8bd7a11268) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240501 08:01:49.620599 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: FlushMRSOp(f80db2e37bbc4716ad3a5200c61ee97b) complete. Timing: real 0.060s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":2320,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1583,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":561,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":139,"threads_started":2,"wal-append.queue_time_us":28074}
I20240501 08:01:49.622143 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling UndoDeltaBlockGCOp(f80db2e37bbc4716ad3a5200c61ee97b): 4839 bytes on disk
I20240501 08:01:49.623430 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: UndoDeltaBlockGCOp(f80db2e37bbc4716ad3a5200c61ee97b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240501 08:01:49.627159 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: FlushMRSOp(7a266f266c3d4fd49fa8b325f1045d61) complete. Timing: real 0.070s user 0.017s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":550,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":2281,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":603,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":175,"threads_started":2,"wal-append.queue_time_us":27046}
I20240501 08:01:49.628425 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling UndoDeltaBlockGCOp(7a266f266c3d4fd49fa8b325f1045d61): 4926 bytes on disk
I20240501 08:01:49.630692 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: UndoDeltaBlockGCOp(7a266f266c3d4fd49fa8b325f1045d61) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240501 08:01:49.633605 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:49.638638 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:49.641507 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:49.641757 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:49.642995 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:49.673826 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:49.680948 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling FlushMRSOp(b4707b952d174a9bb877ad8bd7a11268): perf score=0.033384
I20240501 08:01:49.730952 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:49.734694 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:49.739032 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:49.747860 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:49.749967 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:49.750690 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:49.756103 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: FlushMRSOp(b4707b952d174a9bb877ad8bd7a11268) complete. Timing: real 0.075s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6082,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":2043,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":576,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":147,"threads_started":2,"wal-append.queue_time_us":36422}
I20240501 08:01:49.757205 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling UndoDeltaBlockGCOp(b4707b952d174a9bb877ad8bd7a11268): 4586 bytes on disk
I20240501 08:01:49.757804 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: UndoDeltaBlockGCOp(b4707b952d174a9bb877ad8bd7a11268) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
W20240501 08:01:49.761696 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:49.764708 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:49.766831 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:49.779628 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:49.798520 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:49.803323 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:49.813519 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:49.823351 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:49.897032 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:49.900782 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:49.904093 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:49.911023 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:01:49.911067 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:49.923027 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:49.927553 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:49.980461 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:49.981139 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:49.981671 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:49.984319 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:49.985404 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:50.018625 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:50.019364 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:50.043828 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:50.073385 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:50.079988 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:50.082387 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:50.083165 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:50.083575 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:50.097514 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:50.108438 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:50.163589 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:50.164310 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:50.172713 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:50.172972 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:50.176254 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:50.177655 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:50.209220 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:50.211971 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:50.243721 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:50.252756 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:50.254997 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:50.257355 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:50.261348 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling FlushMRSOp(90f11b5693d7456d8a604e6c8726e69a): perf score=0.033358
I20240501 08:01:50.267017 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:50.272128 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:50.273335 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:50.285369 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: FlushMRSOp(90f11b5693d7456d8a604e6c8726e69a) complete. Timing: real 0.024s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8754,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1377,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":833,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":172,"threads_started":2,"wal-append.queue_time_us":5336}
I20240501 08:01:50.286171 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling UndoDeltaBlockGCOp(90f11b5693d7456d8a604e6c8726e69a): 4926 bytes on disk
I20240501 08:01:50.286643 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: UndoDeltaBlockGCOp(90f11b5693d7456d8a604e6c8726e69a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
W20240501 08:01:50.331449 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:01:50.363304 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:50.369675 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling FlushMRSOp(b21b6e2023e842739a7c0cc946e7c2fd): perf score=0.033387
I20240501 08:01:50.372475 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:50.377549 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:50.378975 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling FlushMRSOp(f61e70ba02384f81b9ae36b1c65465b9): perf score=0.033388
I20240501 08:01:50.383781 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling FlushMRSOp(b21b6e2023e842739a7c0cc946e7c2fd): perf score=0.033396
I20240501 08:01:50.393675 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:50.395017 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:50.402531 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:50.419129 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:50.427425 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: FlushMRSOp(b21b6e2023e842739a7c0cc946e7c2fd) complete. Timing: real 0.043s user 0.015s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":568,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":178,"threads_started":2,"wal-append.queue_time_us":22377}
I20240501 08:01:50.430006 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling UndoDeltaBlockGCOp(b21b6e2023e842739a7c0cc946e7c2fd): 4839 bytes on disk
I20240501 08:01:50.430738 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: UndoDeltaBlockGCOp(b21b6e2023e842739a7c0cc946e7c2fd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240501 08:01:50.446769 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:50.451371 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: FlushMRSOp(f61e70ba02384f81b9ae36b1c65465b9) complete. Timing: real 0.072s user 0.017s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":18481,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":764,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":175,"threads_started":2,"wal-append.queue_time_us":16766}
I20240501 08:01:50.452613 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling UndoDeltaBlockGCOp(f61e70ba02384f81b9ae36b1c65465b9): 4586 bytes on disk
I20240501 08:01:50.454655 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:50.460454 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:50.462886 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:50.471554 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:50.475056 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: FlushMRSOp(b21b6e2023e842739a7c0cc946e7c2fd) complete. Timing: real 0.105s user 0.016s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":13732,"dirs.run_cpu_time_us":246,"dirs.run_wall_time_us":19336,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":687,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":190,"threads_started":2,"wal-append.queue_time_us":30908}
I20240501 08:01:50.476114 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling UndoDeltaBlockGCOp(b21b6e2023e842739a7c0cc946e7c2fd): 4839 bytes on disk
I20240501 08:01:50.477114 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: UndoDeltaBlockGCOp(b21b6e2023e842739a7c0cc946e7c2fd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240501 08:01:50.490788 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: UndoDeltaBlockGCOp(f61e70ba02384f81b9ae36b1c65465b9) complete. Timing: real 0.025s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":24743,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":3}
I20240501 08:01:50.493677 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:50.495582 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:50.578284 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:50.580399 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:50.588629 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:50.592455 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:50.602774 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:50.605290 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:50.605587 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:50.616168 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:50.617208 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:50.623276 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:50.634332 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:50.643308 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:50.666823 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:50.676573 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:50.682780 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling FlushMRSOp(ea97f34590684bcf998228245dbcd9b0): perf score=0.033338
W20240501 08:01:50.693187 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:50.698307 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: FlushMRSOp(ea97f34590684bcf998228245dbcd9b0) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1450,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":539,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":88,"threads_started":1,"wal-append.queue_time_us":182}
I20240501 08:01:50.699496 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling UndoDeltaBlockGCOp(ea97f34590684bcf998228245dbcd9b0): 4839 bytes on disk
I20240501 08:01:50.700173 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: UndoDeltaBlockGCOp(ea97f34590684bcf998228245dbcd9b0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240501 08:01:50.728713 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling FlushMRSOp(164fd265561e47128e7ff585b8cfab07): perf score=0.033358
I20240501 08:01:50.742369 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling FlushMRSOp(5b7de08d2f004cd8abc2bbe1cf7b1444): perf score=0.033361
I20240501 08:01:50.747565 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: FlushMRSOp(164fd265561e47128e7ff585b8cfab07) complete. Timing: real 0.019s user 0.012s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":46,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":516,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":81,"threads_started":1,"wal-append.queue_time_us":171}
I20240501 08:01:50.749632 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling FlushMRSOp(ea97f34590684bcf998228245dbcd9b0): perf score=0.033363
I20240501 08:01:50.767665 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:50.781762 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:50.782862 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:50.783113 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: FlushMRSOp(ea97f34590684bcf998228245dbcd9b0) complete. Timing: real 0.033s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1398,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":733,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":77,"threads_started":1,"wal-append.queue_time_us":10415}
I20240501 08:01:50.784036 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling UndoDeltaBlockGCOp(164fd265561e47128e7ff585b8cfab07): 4926 bytes on disk
I20240501 08:01:50.784433 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: UndoDeltaBlockGCOp(164fd265561e47128e7ff585b8cfab07) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240501 08:01:50.785218 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling UndoDeltaBlockGCOp(ea97f34590684bcf998228245dbcd9b0): 4839 bytes on disk
I20240501 08:01:50.785667 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:50.785739 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: UndoDeltaBlockGCOp(ea97f34590684bcf998228245dbcd9b0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 08:01:50.787020 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:50.793265 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:50.795225 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: FlushMRSOp(5b7de08d2f004cd8abc2bbe1cf7b1444) complete. Timing: real 0.053s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1399,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":612,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":76,"threads_started":1,"wal-append.queue_time_us":12797}
I20240501 08:01:50.798190 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling UndoDeltaBlockGCOp(5b7de08d2f004cd8abc2bbe1cf7b1444): 4585 bytes on disk
I20240501 08:01:50.799007 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: UndoDeltaBlockGCOp(5b7de08d2f004cd8abc2bbe1cf7b1444) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":196,"lbm_reads_lt_1ms":4}
W20240501 08:01:50.802676 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:01:50.825196 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:01:50.849028 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:50.883244 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:50.888420 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:50.890302 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:50.894671 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:50.899420 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:50.920909 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:50.930363 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:50.963517 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:50.967411 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:50.975102 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:50.975728 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:50.981122 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:50.981956 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:50.996223 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:51.065670 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:51.065863 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:51.068434 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:51.070439 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:51.083539 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:51.093484 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:51.124771 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:51.125101 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:51.135756 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:51.140239 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:51.148701 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:51.161690 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:51.162389 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:51.164407 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:51.215251 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:51.216663 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:51.220889 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:51.232295 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:51.234689 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:51.235314 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:51.286805 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:01:51.298878 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240501 08:01:51.331737 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:51.336881 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:51.340142 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:51.347301 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:51.350247 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:51.352131 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:51.367012 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:51.374799 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:51.376149 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:51.382151 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:51.385277 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:51.387166 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:51.397189 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:51.399742 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:51.423822 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:51.435178 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:51.453172 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling FlushMRSOp(6d214be86534458091f54a386d7d3334): perf score=0.033369
W20240501 08:01:51.465902 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:51.470492 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: FlushMRSOp(6d214be86534458091f54a386d7d3334) complete. Timing: real 0.017s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3657,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1337,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":606,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":163,"threads_started":2,"wal-append.queue_time_us":309}
I20240501 08:01:51.471400 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling UndoDeltaBlockGCOp(6d214be86534458091f54a386d7d3334): 4586 bytes on disk
I20240501 08:01:51.471879 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: UndoDeltaBlockGCOp(6d214be86534458091f54a386d7d3334) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240501 08:01:51.507493 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:01:51.513063 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:51.521759 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:51.529903 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:51.532584 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:51.542302 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling FlushMRSOp(6d214be86534458091f54a386d7d3334): perf score=0.033397
I20240501 08:01:51.575560 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:51.577095 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:51.578873 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:51.579486 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:51.583818 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:51.595459 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 08:01:51.611955 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:51.616847 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: FlushMRSOp(6d214be86534458091f54a386d7d3334) complete. Timing: real 0.074s user 0.011s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1104,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":1533,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":552,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1124,"threads_started":2,"wal-append.queue_time_us":9492}
I20240501 08:01:51.617897 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling FlushMRSOp(5e8550ea95984d08937b9b3e2af12bec): perf score=0.033417
I20240501 08:01:51.623963 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:51.626410 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:51.635473 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:51.636657 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:51.638015 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:51.647035 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:51.655325 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:51.658703 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:51.674808 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:51.682935 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:51.690634 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: FlushMRSOp(5e8550ea95984d08937b9b3e2af12bec) complete. Timing: real 0.073s user 0.018s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1313,"drs_written":1,"lbm_read_time_us":148,"lbm_reads_lt_1ms":4,"lbm_write_time_us":622,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":94,"threads_started":1,"wal-append.queue_time_us":6300}
I20240501 08:01:51.691634 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling UndoDeltaBlockGCOp(5e8550ea95984d08937b9b3e2af12bec): 4839 bytes on disk
I20240501 08:01:51.703771 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: UndoDeltaBlockGCOp(5e8550ea95984d08937b9b3e2af12bec) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240501 08:01:51.704749 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling UndoDeltaBlockGCOp(6d214be86534458091f54a386d7d3334): 4586 bytes on disk
I20240501 08:01:51.707624 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: UndoDeltaBlockGCOp(6d214be86534458091f54a386d7d3334) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240501 08:01:51.708761 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:51.784505 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:51.790329 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling FlushMRSOp(b4d15e19eed54aefb6b96276fad98324): perf score=0.033351
I20240501 08:01:51.804077 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling FlushMRSOp(ed88ea311f474ab28cc6fbe6069258bf): perf score=0.033350
I20240501 08:01:51.819970 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:51.823832 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:51.834637 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:51.837877 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: FlushMRSOp(ed88ea311f474ab28cc6fbe6069258bf) complete. Timing: real 0.034s user 0.020s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":704,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":303,"threads_started":2,"wal-append.queue_time_us":169}
I20240501 08:01:51.838783 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling UndoDeltaBlockGCOp(ed88ea311f474ab28cc6fbe6069258bf): 4837 bytes on disk
I20240501 08:01:51.839288 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: UndoDeltaBlockGCOp(ed88ea311f474ab28cc6fbe6069258bf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240501 08:01:51.846109 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: FlushMRSOp(b4d15e19eed54aefb6b96276fad98324) complete. Timing: real 0.056s user 0.020s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":11342,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":3197,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2455,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":187,"threads_started":2,"wal-append.queue_time_us":12626}
I20240501 08:01:51.846704 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:51.847524 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling UndoDeltaBlockGCOp(b4d15e19eed54aefb6b96276fad98324): 4926 bytes on disk
I20240501 08:01:51.848018 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: UndoDeltaBlockGCOp(b4d15e19eed54aefb6b96276fad98324) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240501 08:01:51.849135 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling FlushMRSOp(5c47aaeb16c945ae84ece505aada2a7c): perf score=0.033365
I20240501 08:01:51.855962 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:51.857851 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:51.874074 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: FlushMRSOp(5c47aaeb16c945ae84ece505aada2a7c) complete. Timing: real 0.024s user 0.012s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":141,"dirs.run_wall_time_us":1884,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":599,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":73,"threads_started":1,"wal-append.queue_time_us":4587}
I20240501 08:01:51.875102 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling UndoDeltaBlockGCOp(5c47aaeb16c945ae84ece505aada2a7c): 4586 bytes on disk
I20240501 08:01:51.875576 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: UndoDeltaBlockGCOp(5c47aaeb16c945ae84ece505aada2a7c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
W20240501 08:01:51.883062 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:01:51.917646 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:51.960083 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling FlushMRSOp(b4d15e19eed54aefb6b96276fad98324): perf score=0.033398
I20240501 08:01:51.971444 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:51.974035 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:51.978962 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:51.983217 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: FlushMRSOp(b4d15e19eed54aefb6b96276fad98324) complete. Timing: real 0.023s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1844,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":626,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":77,"threads_started":1,"wal-append.queue_time_us":3056}
I20240501 08:01:51.985167 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling UndoDeltaBlockGCOp(b4d15e19eed54aefb6b96276fad98324): 4926 bytes on disk
I20240501 08:01:51.985854 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: UndoDeltaBlockGCOp(b4d15e19eed54aefb6b96276fad98324) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240501 08:01:51.986779 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling FlushMRSOp(ed88ea311f474ab28cc6fbe6069258bf): perf score=0.033404
I20240501 08:01:51.989887 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:51.995843 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:52.000994 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:52.022504 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:52.056541 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: FlushMRSOp(ed88ea311f474ab28cc6fbe6069258bf) complete. Timing: real 0.070s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":79,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1425,"drs_written":1,"lbm_read_time_us":306,"lbm_reads_lt_1ms":4,"lbm_write_time_us":661,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":97,"threads_started":1,"wal-append.queue_time_us":10164}
I20240501 08:01:52.057612 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling UndoDeltaBlockGCOp(ed88ea311f474ab28cc6fbe6069258bf): 4837 bytes on disk
I20240501 08:01:52.058087 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: UndoDeltaBlockGCOp(ed88ea311f474ab28cc6fbe6069258bf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240501 08:01:52.071985 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:52.072453 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:52.072711 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:52.075611 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:52.079030 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:52.082737 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:52.095096 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:52.108564 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:52.150447 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:01:52.162830 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:01:52.174553 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:52.176589 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:52.182834 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:52.184106 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:52.185479 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:52.197192 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:52.215739 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:52.241024 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:52.248904 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:52.252142 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:52.253628 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:52.262066 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:52.271402 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240501 08:01:52.274195 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:52.356318 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:52.361562 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:52.367291 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:52.369974 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:52.376396 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:52.379941 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling FlushMRSOp(3f6e4ae97e884219ac7d0d94997d9cc9): perf score=0.033342
I20240501 08:01:52.416592 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:52.421885 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:52.431401 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: FlushMRSOp(3f6e4ae97e884219ac7d0d94997d9cc9) complete. Timing: real 0.051s user 0.018s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":165,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1528,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":648,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":156,"threads_started":2,"wal-append.queue_time_us":11756}
I20240501 08:01:52.432433 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling FlushMRSOp(7c23e3bd7ec34876956046e5d32e4d82): perf score=0.033354
I20240501 08:01:52.434965 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:52.435248 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:52.444281 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:52.447037 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 08:01:52.447305 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:52.448115 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:52.450631 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:52.475517 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling FlushMRSOp(18b146cc4e164e1d9f9f827217c9b3ab): perf score=0.033367
I20240501 08:01:52.479799 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: FlushMRSOp(7c23e3bd7ec34876956046e5d32e4d82) complete. Timing: real 0.047s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1479,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1076,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":73,"threads_started":1,"wal-append.queue_time_us":8526}
I20240501 08:01:52.480758 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling UndoDeltaBlockGCOp(3f6e4ae97e884219ac7d0d94997d9cc9): 4926 bytes on disk
I20240501 08:01:52.481185 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: UndoDeltaBlockGCOp(3f6e4ae97e884219ac7d0d94997d9cc9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240501 08:01:52.481963 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling UndoDeltaBlockGCOp(7c23e3bd7ec34876956046e5d32e4d82): 4586 bytes on disk
I20240501 08:01:52.482331 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: UndoDeltaBlockGCOp(7c23e3bd7ec34876956046e5d32e4d82) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 08:01:52.494516 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: FlushMRSOp(18b146cc4e164e1d9f9f827217c9b3ab) complete. Timing: real 0.019s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":4025,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":3728,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":476,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":903,"threads_started":2,"wal-append.queue_time_us":880}
I20240501 08:01:52.495748 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling UndoDeltaBlockGCOp(18b146cc4e164e1d9f9f827217c9b3ab): 4840 bytes on disk
I20240501 08:01:52.496783 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: UndoDeltaBlockGCOp(18b146cc4e164e1d9f9f827217c9b3ab) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240501 08:01:52.498476 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:52.531597 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:52.548600 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:52.550580 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:52.553659 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:52.560431 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:01:52.560726 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling FlushMRSOp(18b146cc4e164e1d9f9f827217c9b3ab): perf score=0.033390
W20240501 08:01:52.567456 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:01:52.569916 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:52.570626 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:52.574265 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:52.574285 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:52.592253 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling FlushMRSOp(3f6e4ae97e884219ac7d0d94997d9cc9): perf score=0.033403
I20240501 08:01:52.621575 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: FlushMRSOp(3f6e4ae97e884219ac7d0d94997d9cc9) complete. Timing: real 0.029s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":2184,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":9704,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":517,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":158,"threads_started":2,"wal-append.queue_time_us":1315}
I20240501 08:01:52.621593 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: FlushMRSOp(18b146cc4e164e1d9f9f827217c9b3ab) complete. Timing: real 0.061s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":18335,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":9635,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":703,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":16024,"threads_started":2,"wal-append.queue_time_us":5934}
I20240501 08:01:52.622524 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling UndoDeltaBlockGCOp(3f6e4ae97e884219ac7d0d94997d9cc9): 4926 bytes on disk
I20240501 08:01:52.623085 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling UndoDeltaBlockGCOp(18b146cc4e164e1d9f9f827217c9b3ab): 4840 bytes on disk
I20240501 08:01:52.623160 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: UndoDeltaBlockGCOp(3f6e4ae97e884219ac7d0d94997d9cc9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240501 08:01:52.623497 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: UndoDeltaBlockGCOp(18b146cc4e164e1d9f9f827217c9b3ab) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 08:01:52.638765 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:52.639201 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:52.642119 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:52.642963 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:52.646384 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:52.656211 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:52.672067 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:01:52.737515 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:52.782614 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:52.790599 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:52.794133 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:52.804919 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:52.807947 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:52.823637 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:52.825284 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:52.828825 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:52.832013 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:52.837136 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:52.837661 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:52.839481 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:52.881314 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:52.968056 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:52.979061 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:52.980696 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:52.997669 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:53.001168 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:53.024657 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:53.029191 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:53.031602 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:53.034690 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:53.036163 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:53.049033 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:53.051299 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:53.065810 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:53.068881 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:53.076521 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:53.082768 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:53.085734 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:53.102547 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:53.106878 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:53.109889 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:53.110741 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:53.131490 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:53.142051 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:53.230330 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:53.235117 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:53.247334 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:53.251858 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:53.264629 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:53.272500 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:53.282786 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:53.283555 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:53.290001 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:53.290618 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:53.296309 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:53.312549 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:53.348397 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:01:53.364189 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:01:53.432571 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:53.445789 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:53.449102 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:53.453872 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:53.480731 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:53.481274 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:53.485205 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:53.488857 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:53.504249 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:53.510493 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:53.518851 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:53.528846 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:53.529516 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:53.544663 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:53.548125 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:53.548655 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:53.567041 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:53.568746 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:53.571249 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:53.596827 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:53.606298 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:53.679646 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:01:53.680604 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:53.698280 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:53.705513 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:53.708213 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:53.708779 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:53.710412 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:53.715675 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:53.829867 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:01:53.832150 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:01:53.832965 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:53.833125 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:53.839709 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:53.840622 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:53.850762 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:53.852619 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:53.860064 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:53.866026 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:53.879472 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:53.947607 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:53.951813 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:53.954376 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:53.957139 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:53.971729 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:53.987453 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:54.012993 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:54.014220 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:54.037989 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:54.040174 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:54.040362 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:54.042826 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:54.084491 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:01:54.090656 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:54.090886 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:54.099443 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:54.102455 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:54.115437 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:54.134088 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:54.141899 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:54.179549 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:01:54.182974 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:54.186230 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:54.188377 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:54.188980 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:54.189323 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:54.196190 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:54.197578 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:54.210366 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:54.212782 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:54.315199 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:54.321540 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:54.323063 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:54.323482 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:54.332700 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:54.360587 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:54.363421 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:54.364177 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:54.373920 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:54.374007 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:54.389361 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:54.413836 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:54.419386 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:54.422695 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:54.424933 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:54.434082 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:01:54.436837 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:54.456494 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:54.468223 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:01:54.471431 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:54.568691 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:54.568733 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:54.569566 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:54.569835 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:54.582372 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:54.583278 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:54.594038 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:54.610741 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:01:54.722124 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:54.728474 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:54.739919 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:54.742211 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:54.745551 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:54.758100 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:54.773483 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:54.789903 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:54.807413 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:54.808174 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:54.809056 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:54.816387 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:54.826390 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:54.839932 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:54.845736 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:54.908926 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:54.910387 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:54.912360 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:54.915712 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:54.928205 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:54.928385 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:54.930760 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:54.940246 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:54.942867 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:54.942996 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:54.943239 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:54.947788 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:54.949748 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:54.964573 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:01:54.970803 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:01:54.976703 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:01:55.018713 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:01:55.059412 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:55.060551 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:55.064641 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:55.067812 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:55.073012 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:55.091025 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:55.106788 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:55.141039 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:55.146414 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:55.155570 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:55.158125 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:55.174801 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:55.207999 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:55.216858 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:01:55.222223 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:55.223791 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:55.234483 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:55.239354 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:55.246233 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:55.247680 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:55.250207 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:55.257416 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:01:55.324542 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:55.327284 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:55.328837 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:55.340312 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:55.343904 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:01:55.349956 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:55.374553 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:55.403834 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:55.404017 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:55.415627 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:55.416431 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:55.423225 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:55.425045 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:55.522775 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:55.524695 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:01:55.525533 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:55.531563 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:55.534261 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:55.542688 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:55.555510 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:55.576349 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:55.589005 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:01:55.591816 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:55.597554 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:55.600224 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:55.609356 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:55.613673 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:55.625886 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:55.645222 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:01:55.709578 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:01:55.727540 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:01:55.744988 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:55.769851 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:55.772225 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:55.778622 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:55.785459 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:55.787076 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:55.788597 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:55.788734 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:55.801802 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:01:55.802151 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:55.806643 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:55.829375 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:01:55.840454 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:55.858659 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:55.858989 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:55.869264 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:55.869822 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:55.870741 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:55.882890 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:55.883236 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:55.989660 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:01:56.004050 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:56.011510 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:56.014487 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:56.015286 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:56.019706 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:56.025421 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:56.033497 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:56.037434 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:56.043381 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:56.050671 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:56.051939 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:56.052659 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:56.078449 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:56.120687 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:01:56.149315 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling FlushMRSOp(f2e63d24d7df4fa3b2b26695e70ebe9f): perf score=0.033334
I20240501 08:01:56.193418 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: FlushMRSOp(f2e63d24d7df4fa3b2b26695e70ebe9f) complete. Timing: real 0.044s user 0.032s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":3642,"dirs.queue_time_us":10526,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":1186,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":600,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":305,"threads_started":3,"wal-append.queue_time_us":8957}
I20240501 08:01:56.194977 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling UndoDeltaBlockGCOp(f2e63d24d7df4fa3b2b26695e70ebe9f): 4922 bytes on disk
I20240501 08:01:56.195741 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: UndoDeltaBlockGCOp(f2e63d24d7df4fa3b2b26695e70ebe9f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240501 08:01:56.215584 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:56.221518 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:56.227370 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:56.232141 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling FlushMRSOp(93602521de174997bda6702d25aafce1): perf score=0.033349
I20240501 08:01:56.232756 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:56.240456 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:01:56.249316 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:56.276258 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:56.304091 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:56.304813 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:56.318899 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:56.320116 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:56.322746 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:56.326783 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240501 08:01:56.341406 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:56.341958 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:56.343441 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:56.373255 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:56.388521 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:56.399739 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:56.419705 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:56.423079 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: FlushMRSOp(93602521de174997bda6702d25aafce1) complete. Timing: real 0.190s user 0.037s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":56457,"dirs.queue_time_us":24453,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":1456,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":618,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":358,"threads_started":3,"wal-append.queue_time_us":7880}
I20240501 08:01:56.424778 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling FlushMRSOp(25cee7e51c7c438aa721d71fb82dc71e): perf score=0.033355
I20240501 08:01:56.438575 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:56.453539 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:56.466188 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: FlushMRSOp(25cee7e51c7c438aa721d71fb82dc71e) complete. Timing: real 0.041s user 0.019s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1712,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4490,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":111,"threads_started":1,"wal-append.queue_time_us":8017}
I20240501 08:01:56.467222 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling UndoDeltaBlockGCOp(93602521de174997bda6702d25aafce1): 4923 bytes on disk
I20240501 08:01:56.467957 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: UndoDeltaBlockGCOp(93602521de174997bda6702d25aafce1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240501 08:01:56.468757 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling UndoDeltaBlockGCOp(25cee7e51c7c438aa721d71fb82dc71e): 4836 bytes on disk
I20240501 08:01:56.469192 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: UndoDeltaBlockGCOp(25cee7e51c7c438aa721d71fb82dc71e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
W20240501 08:01:56.495133 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:01:56.508927 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:01:56.563977 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:56.564873 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:56.566551 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:56.567145 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:56.577394 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:56.578332 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:56.613348 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:01:56.614807 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:01:56.617744 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:01:56.618510 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:56.703086 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:56.713502 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:56.717458 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:56.726173 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:56.731237 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:56.745028 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:56.766703 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:01:56.801776 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:01:56.830878 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:56.831022 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:56.832613 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:56.837531 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:56.846585 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:56.851904 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:56.862594 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:56.867312 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:56.882579 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:56.886974 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:56.895953 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:56.898480 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:56.899176 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:56.948887 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling FlushMRSOp(b9431cf5853c4148ac38855b1c083d4f): perf score=0.033358
W20240501 08:01:56.958447 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:56.959865 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:56.963735 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:56.971570 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling FlushMRSOp(f80db2e37bbc4716ad3a5200c61ee97b): perf score=0.033388
I20240501 08:01:56.973654 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:56.978380 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:56.985100 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:56.987478 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:57.006083 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: FlushMRSOp(f80db2e37bbc4716ad3a5200c61ee97b) complete. Timing: real 0.034s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":171,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1320,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":531,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":154,"threads_started":2,"wal-append.queue_time_us":140}
I20240501 08:01:57.008196 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling FlushMRSOp(5b7de08d2f004cd8abc2bbe1cf7b1444): perf score=0.033347
I20240501 08:01:57.008621 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:57.015327 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: FlushMRSOp(b9431cf5853c4148ac38855b1c083d4f) complete. Timing: real 0.066s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":12382,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":33776,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":599,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":144,"threads_started":2,"wal-append.queue_time_us":173}
I20240501 08:01:57.016278 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling UndoDeltaBlockGCOp(b9431cf5853c4148ac38855b1c083d4f): 4836 bytes on disk
I20240501 08:01:57.016669 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: UndoDeltaBlockGCOp(b9431cf5853c4148ac38855b1c083d4f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240501 08:01:57.017385 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling FlushMRSOp(867ba08a15b44d4c9e6fa92b0d518d1d): perf score=0.033368
I20240501 08:01:57.027272 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: FlushMRSOp(5b7de08d2f004cd8abc2bbe1cf7b1444) complete. Timing: real 0.018s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1278,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":543,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":119,"threads_started":1,"wal-append.queue_time_us":210}
I20240501 08:01:57.028277 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling UndoDeltaBlockGCOp(f80db2e37bbc4716ad3a5200c61ee97b): 4839 bytes on disk
I20240501 08:01:57.028736 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: UndoDeltaBlockGCOp(f80db2e37bbc4716ad3a5200c61ee97b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240501 08:01:57.029458 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling UndoDeltaBlockGCOp(5b7de08d2f004cd8abc2bbe1cf7b1444): 4585 bytes on disk
I20240501 08:01:57.029806 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: UndoDeltaBlockGCOp(5b7de08d2f004cd8abc2bbe1cf7b1444) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240501 08:01:57.033228 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: FlushMRSOp(867ba08a15b44d4c9e6fa92b0d518d1d) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":125,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1646,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":611,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":104,"threads_started":1,"wal-append.queue_time_us":186}
I20240501 08:01:57.033941 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling FlushMRSOp(90f11b5693d7456d8a604e6c8726e69a): perf score=0.033369
I20240501 08:01:57.047772 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: FlushMRSOp(90f11b5693d7456d8a604e6c8726e69a) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1332,"drs_written":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4,"lbm_write_time_us":430,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":96,"threads_started":1,"wal-append.queue_time_us":348}
I20240501 08:01:57.048743 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling UndoDeltaBlockGCOp(90f11b5693d7456d8a604e6c8726e69a): 4926 bytes on disk
I20240501 08:01:57.049147 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: UndoDeltaBlockGCOp(90f11b5693d7456d8a604e6c8726e69a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240501 08:01:57.049929 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling UndoDeltaBlockGCOp(867ba08a15b44d4c9e6fa92b0d518d1d): 4583 bytes on disk
I20240501 08:01:57.050315 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: UndoDeltaBlockGCOp(867ba08a15b44d4c9e6fa92b0d518d1d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240501 08:01:57.056926 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:01:57.063257 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:57.082331 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:57.087574 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:57.088145 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:57.088382 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:57.088936 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:57.089916 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:57.154253 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:01:57.154619 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:01:57.209815 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:57.215750 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:57.216321 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:57.223722 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:57.229465 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:57.242136 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:57.246205 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:57.258975 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:57.272045 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:57.272833 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:57.284389 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:57.289223 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:57.290125 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:57.296965 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:57.353544 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:01:57.354452 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:01:57.385169 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:57.389060 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:57.398339 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:57.404870 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:57.409165 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:57.425843 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:57.426024 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:57.449272 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:57.450608 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:57.459146 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:57.459878 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:57.477428 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:57.493373 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:57.532434 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling FlushMRSOp(ea97f34590684bcf998228245dbcd9b0): perf score=0.033387
W20240501 08:01:57.538887 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 08:01:57.540037 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:57.541991 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:57.549250 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:57.550846 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:57.552748 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling FlushMRSOp(164fd265561e47128e7ff585b8cfab07): perf score=0.033343
I20240501 08:01:57.566502 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:57.568104 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:57.581279 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: FlushMRSOp(164fd265561e47128e7ff585b8cfab07) complete. Timing: real 0.028s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":659,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":152,"threads_started":2,"wal-append.queue_time_us":3269}
I20240501 08:01:57.581509 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:57.582441 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling UndoDeltaBlockGCOp(164fd265561e47128e7ff585b8cfab07): 4926 bytes on disk
I20240501 08:01:57.582903 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: UndoDeltaBlockGCOp(164fd265561e47128e7ff585b8cfab07) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240501 08:01:57.600342 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: FlushMRSOp(ea97f34590684bcf998228245dbcd9b0) complete. Timing: real 0.068s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":25292,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1360,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":607,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":152,"threads_started":2,"wal-append.queue_time_us":157}
I20240501 08:01:57.601469 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling FlushMRSOp(b4d15e19eed54aefb6b96276fad98324): perf score=0.033376
I20240501 08:01:57.616833 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: FlushMRSOp(b4d15e19eed54aefb6b96276fad98324) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1487,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":442,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":81,"threads_started":1,"wal-append.queue_time_us":154}
I20240501 08:01:57.618053 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling UndoDeltaBlockGCOp(b4d15e19eed54aefb6b96276fad98324): 4926 bytes on disk
I20240501 08:01:57.618825 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: UndoDeltaBlockGCOp(b4d15e19eed54aefb6b96276fad98324) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240501 08:01:57.619438 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling UndoDeltaBlockGCOp(ea97f34590684bcf998228245dbcd9b0): 4839 bytes on disk
I20240501 08:01:57.619747 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: UndoDeltaBlockGCOp(ea97f34590684bcf998228245dbcd9b0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240501 08:01:57.646891 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:01:57.668339 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:57.691367 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:57.698580 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:57.699113 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:57.700284 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:57.715234 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:57.721603 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:57.735567 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:57.737221 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:57.753121 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:57.753401 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:57.787398 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:57.794576 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:57.797233 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:57.805559 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:57.808267 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:57.809670 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:57.809779 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:57.810062 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:57.819278 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:57.834362 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling FlushMRSOp(3f6e4ae97e884219ac7d0d94997d9cc9): perf score=0.033396
I20240501 08:01:57.842123 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:57.852300 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:57.864233 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: FlushMRSOp(3f6e4ae97e884219ac7d0d94997d9cc9) complete. Timing: real 0.025s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":159,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1339,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":525,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":83,"threads_started":1,"wal-append.queue_time_us":368}
I20240501 08:01:57.865223 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling UndoDeltaBlockGCOp(3f6e4ae97e884219ac7d0d94997d9cc9): 4926 bytes on disk
I20240501 08:01:57.865694 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: UndoDeltaBlockGCOp(3f6e4ae97e884219ac7d0d94997d9cc9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240501 08:01:57.866484 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling FlushMRSOp(53193305f868433c852755e8a8a81766): perf score=0.033362
I20240501 08:01:57.871060 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling FlushMRSOp(f61e70ba02384f81b9ae36b1c65465b9): perf score=0.033401
I20240501 08:01:57.890132 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: FlushMRSOp(f61e70ba02384f81b9ae36b1c65465b9) complete. Timing: real 0.019s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":1275,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":471,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":89,"threads_started":1,"wal-append.queue_time_us":189}
I20240501 08:01:57.890993 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling UndoDeltaBlockGCOp(f61e70ba02384f81b9ae36b1c65465b9): 4586 bytes on disk
I20240501 08:01:57.892262 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: UndoDeltaBlockGCOp(f61e70ba02384f81b9ae36b1c65465b9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240501 08:01:57.906280 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: FlushMRSOp(53193305f868433c852755e8a8a81766) complete. Timing: real 0.040s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1270,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1309,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":674,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":113,"threads_started":1,"wal-append.queue_time_us":223}
I20240501 08:01:57.907287 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling UndoDeltaBlockGCOp(53193305f868433c852755e8a8a81766): 4926 bytes on disk
I20240501 08:01:57.907719 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: UndoDeltaBlockGCOp(53193305f868433c852755e8a8a81766) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
W20240501 08:01:57.929872 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:57.962466 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:57.964123 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:57.972371 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:57.973416 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:57.973528 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:57.995254 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:58.013955 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:58.014407 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:58.016695 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:58.018642 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:58.025146 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:58.042361 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:58.046026 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:58.046278 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:58.065569 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:58.137238 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:58.139915 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:58.143741 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling FlushMRSOp(48b0ff0769dc426190f038811155c741): perf score=0.033363
I20240501 08:01:58.145496 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:58.147004 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:58.150103 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:58.164840 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: FlushMRSOp(48b0ff0769dc426190f038811155c741) complete. Timing: real 0.021s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":94,"dirs.run_cpu_time_us":238,"dirs.run_wall_time_us":1492,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":592,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":86,"threads_started":1,"wal-append.queue_time_us":174}
I20240501 08:01:58.171255 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling UndoDeltaBlockGCOp(48b0ff0769dc426190f038811155c741): 4836 bytes on disk
I20240501 08:01:58.172542 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: UndoDeltaBlockGCOp(48b0ff0769dc426190f038811155c741) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240501 08:01:58.199028 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:58.200167 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:58.206624 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:58.217835 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:58.227843 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:58.228076 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:58.232300 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:58.248962 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:01:58.262290 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:58.262323 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:58.335639 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:58.342290 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:58.347074 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:58.354815 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:58.367002 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:58.374315 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:58.380023 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:58.389043 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:58.392045 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:58.409145 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:58.412221 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:58.413162 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:58.424921 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:58.426471 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:58.483863 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:58.485095 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:58.486207 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:58.488279 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:58.491505 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:58.494386 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:58.521093 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:58.532891 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:58.602337 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:58.604152 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:58.607683 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:58.609944 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:58.611646 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:58.620497 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:58.621502 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:58.657289 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:58.664839 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling FlushMRSOp(5c47aaeb16c945ae84ece505aada2a7c): perf score=0.033359
I20240501 08:01:58.675029 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling FlushMRSOp(17bb1ff829bc47a6a0bd6030517652aa): perf score=0.033353
W20240501 08:01:58.688366 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:58.692242 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: FlushMRSOp(5c47aaeb16c945ae84ece505aada2a7c) complete. Timing: real 0.027s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2767,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":2035,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":529,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":198,"threads_started":2,"wal-append.queue_time_us":10645}
I20240501 08:01:58.692922 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: FlushMRSOp(17bb1ff829bc47a6a0bd6030517652aa) complete. Timing: real 0.018s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":204,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":1591,"drs_written":1,"lbm_read_time_us":149,"lbm_reads_lt_1ms":4,"lbm_write_time_us":510,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":154,"threads_started":2,"wal-append.queue_time_us":151}
I20240501 08:01:58.693183 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling FlushMRSOp(b4707b952d174a9bb877ad8bd7a11268): perf score=0.033345
I20240501 08:01:58.693710 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling UndoDeltaBlockGCOp(17bb1ff829bc47a6a0bd6030517652aa): 4923 bytes on disk
I20240501 08:01:58.694160 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: UndoDeltaBlockGCOp(17bb1ff829bc47a6a0bd6030517652aa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240501 08:01:58.716593 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: FlushMRSOp(b4707b952d174a9bb877ad8bd7a11268) complete. Timing: real 0.023s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1373,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":570,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":91,"threads_started":1,"wal-append.queue_time_us":175}
I20240501 08:01:58.717584 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling UndoDeltaBlockGCOp(5c47aaeb16c945ae84ece505aada2a7c): 4586 bytes on disk
I20240501 08:01:58.718538 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: UndoDeltaBlockGCOp(5c47aaeb16c945ae84ece505aada2a7c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240501 08:01:58.719409 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling UndoDeltaBlockGCOp(b4707b952d174a9bb877ad8bd7a11268): 4586 bytes on disk
I20240501 08:01:58.719955 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: UndoDeltaBlockGCOp(b4707b952d174a9bb877ad8bd7a11268) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240501 08:01:58.761595 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:58.763967 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:58.778801 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:58.784462 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:58.795439 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:58.796800 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:58.798739 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:58.802011 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:58.804036 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:58.815922 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:58.848135 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:58.850920 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:58.851938 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:01:58.853175 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:01:58.880888 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:01:58.938812 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:58.940940 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:58.950338 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:58.950968 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:58.968895 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:58.973839 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:58.974886 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:59.010706 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:59.010996 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:59.013536 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:59.016911 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:59.018627 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:59.028046 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:01:59.028410 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:01:59.088615 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:01:59.107504 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:01:59.123739 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:59.126605 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:59.127151 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:59.130612 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:59.135768 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:59.143589 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:59.147460 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:01:59.202399 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:01:59.222249 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling FlushMRSOp(a24be92aec7544089c7818f2802b9c85): perf score=0.033339
I20240501 08:01:59.230768 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:59.237215 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:59.255939 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:59.258440 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:59.265741 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:59.280915 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:59.281131 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:59.285497 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: FlushMRSOp(a24be92aec7544089c7818f2802b9c85) complete. Timing: real 0.063s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3943,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":24572,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2887,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":154,"threads_started":2,"wal-append.queue_time_us":269}
I20240501 08:01:59.289268 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling UndoDeltaBlockGCOp(a24be92aec7544089c7818f2802b9c85): 4839 bytes on disk
I20240501 08:01:59.289693 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:59.294348 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:59.295753 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: UndoDeltaBlockGCOp(a24be92aec7544089c7818f2802b9c85) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240501 08:01:59.306510 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:59.314224 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:59.334782 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:01:59.367743 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:01:59.390237 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:59.394201 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:59.394909 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:59.394941 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:59.411310 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:59.447050 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling FlushMRSOp(a24be92aec7544089c7818f2802b9c85): perf score=0.033393
I20240501 08:01:59.449517 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:59.451673 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:59.466689 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:59.466825 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:59.467718 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:59.468596 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:01:59.470603 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:01:59.479640 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:01:59.482189 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:59.492184 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: FlushMRSOp(a24be92aec7544089c7818f2802b9c85) complete. Timing: real 0.045s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":24107,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":8295,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":526,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":159,"threads_started":2,"wal-append.queue_time_us":186}
I20240501 08:01:59.493166 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling UndoDeltaBlockGCOp(a24be92aec7544089c7818f2802b9c85): 4839 bytes on disk
I20240501 08:01:59.493659 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: UndoDeltaBlockGCOp(a24be92aec7544089c7818f2802b9c85) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240501 08:01:59.514896 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:01:59.595122 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:59.597090 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:59.609764 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:59.623731 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:59.628657 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:59.639890 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:59.651005 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:59.663345 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:59.671742 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:59.675338 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:59.680893 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:59.687877 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:01:59.721191 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:01:59.736037 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:01:59.744352 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:01:59.767879 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:01:59.769703 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:01:59.774052 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:01:59.807780 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:59.807987 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:59.809257 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:59.813457 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:59.816731 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:59.824304 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:01:59.886843 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:59.891855 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:59.892328 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:01:59.905157 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:59.909173 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:01:59.909253 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:01:59.911578 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:59.963236 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:01:59.968088 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:01:59.971820 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:01:59.982985 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:01:59.993441 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:00.004674 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:00.031376 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:00.058116 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:00.058329 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:00.071009 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:00.075744 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:00.077190 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:00.082526 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:00.138635 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:02:00.153040 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:00.160398 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:00.165622 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:00.175442 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:00.182228 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:00.184036 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:00.199087 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:02:00.200721 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:02:00.208348 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:00.212692 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:00.225296 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:00.254654 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:00.257318 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:00.267140 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:00.268009 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:00.269205 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:00.277285 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:00.282994 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:00.285430 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:00.291589 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:00.294317 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:00.304271 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:00.305665 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:00.354678 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:00.408411 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:00.409848 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:00.411540 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:00.412443 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:00.413939 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:00.421736 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:00.452172 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:02:00.520939 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:00.639430 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:00.640125 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:00.640753 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:00.643648 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:00.660271 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:00.661079 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:00.664502 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:00.668040 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:00.680820 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:00.705044 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:00.705999 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:00.713546 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:00.727325 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:00.728029 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:00.742754 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:00.745218 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:00.762166 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:00.819553 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:00.819564 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:00.831221 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:00.831866 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:00.832019 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:00.855293 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:00.909819 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:00.912776 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:00.917711 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:00.918134 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:00.927124 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:00.927521 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:00.954140 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:00.966992 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:02:00.985195 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:01.019181 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:01.021360 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:01.022246 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:01.023386 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:01.047888 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:01.056532 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:01.090826 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:01.099084 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:01.099565 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:01.120272 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:01.121640 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:01.128038 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:01.165350 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:01.170051 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:01.177698 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:01.186820 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:01.196084 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:01.196134 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:01.198374 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:01.212391 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:01.254786 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:01.256451 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:01.258085 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:01.258117 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:01.258965 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:01.263815 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:01.270960 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:01.276302 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:01.329017 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 08:02:01.364980 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:01.371827 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:01.378281 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:01.379557 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:01.383680 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:01.392181 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:01.400800 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:01.434410 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:01.436290 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:01.442343 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:01.444588 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:01.445215 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:01.475775 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:01.519814 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:01.543967 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:01.551476 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:01.553270 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:01.573215 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:01.573902 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:01.600272 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:01.601487 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:01.612013 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:01.612180 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:01.629081 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:01.629778 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:01.636162 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:01.644346 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:01.698401 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:02:01.699493 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:01.706193 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:01.708091 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:01.712850 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:01.723359 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:01.726648 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:01.728035 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:01.752959 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:01.800992 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:01.803464 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:01.806885 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:01.807512 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:01.809492 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:02:01.811981 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:01.815728 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:01.820219 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:01.833012 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:01.897377 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:01.897491 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:01.902276 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:01.904191 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:01.922019 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:01.931324 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:01.991011 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:01.994597 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:02.006664 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:02.010634 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:02.021245 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:02.024955 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:02.028687 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:02.044097 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:02.049780 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:02.061710 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:02.076654 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:02.089867 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:02.090763 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:02.101001 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:02.128335 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:02.133888 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:02.136685 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:02.142575 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:02.147272 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:02.148658 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:02.161732 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:02.216594 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:02.252707 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:02.252821 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:02.256290 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:02.257174 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:02.261723 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:02.269870 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:02.287398 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:02.297605 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:02:02.335245 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:02.362044 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:02:02.362982 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:02.377722 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:02.393625 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:02.401055 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:02.405020 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:02.426950 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:02.428239 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:02.430229 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:02.432585 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:02.433400 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:02.443358 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:02.451978 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:02.458572 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:02.463431 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:02.473093 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:02.480293 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:02:02.484158 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:02:02.578771 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:02.584321 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:02.584609 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:02.593255 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:02.596000 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:02.597765 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:02.618564 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:02.622803 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:02.627184 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:02.628727 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:02.634999 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:02.638216 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:02.653851 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:02.679482 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:02.783360 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:02.785470 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:02.799360 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:02.801697 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:02.805028 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:02.813144 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:02.822710 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:02.823328 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:02.859263 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:02.867336 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:02.872035 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:02.893508 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:02.895062 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:02.895331 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:02.897282 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:02.897972 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:02.899995 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:02.912312 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:02.918949 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:02.922855 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:02.969295 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:02.973791 11598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:02:02.984815 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:03.004391 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:03.018965 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:03.021306 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:03.025987 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:03.036589 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:03.047147 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:03.070811 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:03.083036 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:03.084195 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:03.093240 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:03.093312 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:03.105067 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:03.124378 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:03.127301 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:03.195237 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:03.199115 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:03.198652 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:03.202343 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:03.202649 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:03.205890 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:03.217790 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:03.278681 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:03.284379 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:03.289462 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:03.289539 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:03.295320 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:03.295419 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:03.295938 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:03.296125 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:03.391672 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:03.397009 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:03.399551 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:03.402016 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:03.402779 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:03.423413 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:03.452461 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:03.460269 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:03.468897 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:03.483155 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:03.496837 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:03.498347 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:03.502919 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:02:03.516708 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:03.516731 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:03.529940 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:03.538607 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:03.540825 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:03.548480 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:03.549043 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:03.562280 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:03.580901 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:03.653362 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:03.661134 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:03.664871 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:03.670076 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:03.677733 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:03.688247 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:03.693364 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:03.700034 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:03.707610 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:03.733075 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:03.742370 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:03.743968 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:03.760819 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling FlushMRSOp(7c23e3bd7ec34876956046e5d32e4d82): perf score=0.033364
I20240501 08:02:03.769979 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:03.784611 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: FlushMRSOp(7c23e3bd7ec34876956046e5d32e4d82) complete. Timing: real 0.024s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":187,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":6793,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":587,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":151,"threads_started":2,"wal-append.queue_time_us":166}
I20240501 08:02:03.785589 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling UndoDeltaBlockGCOp(7c23e3bd7ec34876956046e5d32e4d82): 4586 bytes on disk
I20240501 08:02:03.786026 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: UndoDeltaBlockGCOp(7c23e3bd7ec34876956046e5d32e4d82) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
W20240501 08:02:03.812654 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:03.874689 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:03.883450 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:03.884207 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:03.884685 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:03.915011 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:03.917227 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:03.921126 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:03.925663 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:03.931711 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:03.939716 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:03.951040 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:03.966202 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:04.021440 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:04.044234 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:02:04.047071 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:04.058655 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:04.063495 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:04.068112 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:04.073199 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:04.077452 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:04.087879 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:04.092216 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:04.094957 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:04.098287 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:04.100340 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:04.110831 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:04.118471 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:04.140893 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:04.141557 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:04.148998 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:04.159319 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:02:04.220018 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:04.246548 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:04.246549 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:04.248333 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:04.257508 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:04.263906 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:04.295711 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:04.296550 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:04.297397 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:04.308948 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:04.310241 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:04.315433 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:04.320605 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:04.330610 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:04.378050 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:04.446619 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:04.446988 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:04.450548 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:04.457551 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:04.459115 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:04.459501 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:02:04.459805 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:04.503077 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:04.506319 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:04.510378 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:04.513180 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:04.536804 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:02:04.537791 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:04.539108 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:04.616343 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:04.650403 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:04.664829 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:04.665453 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:04.675585 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:04.676893 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:04.688285 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:04.698845 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:04.712500 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:04.721936 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:04.732795 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:04.739346 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:02:04.739432 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:04.749231 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:04.763162 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:04.765628 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:04.783690 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:04.785678 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:04.788602 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:04.794663 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:04.794839 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:04.804878 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:04.815472 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:04.835369 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:04.900452 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:04.921048 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:04.928078 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:04.930500 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:02:04.935418 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:04.940443 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:04.956009 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:04.970439 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:04.980762 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:04.984295 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:04.984572 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:04.985739 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:04.991851 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:05.007001 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:05.051210 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:05.109194 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:05.110224 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:05.111548 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:05.125875 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:05.130657 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:05.163504 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:05.176671 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:05.178876 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:05.179246 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:05.190927 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:05.193957 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:05.207221 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:05.225410 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:05.238029 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:05.240532 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:02:05.241950 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:05.246392 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:05.247514 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:05.249084 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:02:05.250314 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:05.256289 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:05.262454 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:05.269590 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:05.377148 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:05.377427 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:05.380661 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:02:05.382328 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:05.383119 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:05.390540 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:05.398818 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:05.404871 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:05.407577 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:05.410010 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:05.420544 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:05.425686 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:05.436622 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:05.448822 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:02:05.452152 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:05.556893 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:05.560535 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:05.561317 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:05.572927 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:05.583592 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:05.583689 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:05.622799 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:02:05.632941 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:05.635169 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:05.635771 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:05.640960 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:05.657184 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:05.671924 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:05.709405 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:05.709869 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:05.730129 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:05.759107 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:05.763815 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:05.768822 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:05.770658 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:05.771437 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:05.773257 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:05.773407 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:05.777474 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:02:05.808131 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:05.809089 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:05.810381 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:05.823046 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:05.824383 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:05.825547 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:05.903735 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:05.928987 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:02:05.951388 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:05.956995 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:05.969159 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:05.976529 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:05.979014 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:05.978993 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:05.987056 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:05.997114 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:06.013173 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:06.013680 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:06.015094 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:06.055954 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:06.075984 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling FlushMRSOp(f80db2e37bbc4716ad3a5200c61ee97b): perf score=0.033399
W20240501 08:02:06.087339 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:02:06.090339 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:06.099675 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:06.100054 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:06.115787 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:06.116989 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:06.116984 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:06.118618 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: FlushMRSOp(f80db2e37bbc4716ad3a5200c61ee97b) complete. Timing: real 0.042s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":13499,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":599,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":4796}
I20240501 08:02:06.119522 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling UndoDeltaBlockGCOp(f80db2e37bbc4716ad3a5200c61ee97b): 4839 bytes on disk
W20240501 08:02:06.119729 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:02:06.130682 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: UndoDeltaBlockGCOp(f80db2e37bbc4716ad3a5200c61ee97b) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":178,"lbm_reads_lt_1ms":4}
I20240501 08:02:06.202693 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:06.203858 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:06.205214 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:06.206180 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:06.208801 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:06.219798 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:06.293828 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:06.342872 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:06.345387 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:06.356701 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:06.360934 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:06.361222 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:06.366626 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:06.400110 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:06.424928 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:06.425611 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:06.433353 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:06.438292 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:06.450096 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:06.460815 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:06.505692 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:02:06.506551 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:06.508786 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:06.509411 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:06.512914 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:06.514209 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:06.514343 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:06.515187 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:06.618476 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:06.620761 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:06.629328 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:06.629335 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:06.638641 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:06.647130 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:06.667019 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:06.670905 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:06.676729 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:06.677228 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:06.686136 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:06.692437 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:06.702296 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:06.731154 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:06.739943 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:06.746528 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:06.788372 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:02:06.789076 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:06.805104 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:02:06.807071 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:06.814713 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:06.815584 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:06.817615 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:06.823710 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:06.849938 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:06.863036 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:06.871881 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:02:06.873272 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:06.874101 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:06.887434 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:06.900800 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:06.919478 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:06.961412 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:06.965792 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:06.966570 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:06.967037 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:06.973464 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:07.001875 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:07.040234 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:07.075176 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:07.077158 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:07.077771 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:07.082888 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:07.086583 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:07.087098 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:07.092742 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:07.094756 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:07.094933 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:07.095381 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:07.106947 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:07.110152 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:07.122406 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:07.124015 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:07.130805 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:07.161430 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:07.205437 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:07.239269 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:07.239411 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:07.250438 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:07.251200 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:07.253626 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:07.277518 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:07.289184 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:07.332844 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:07.334635 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:07.338614 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:07.339702 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:07.340497 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:07.350967 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:07.368885 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:07.384333 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:02:07.385327 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:07.424737 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:07.427361 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:07.441874 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:07.447402 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:07.461627 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:07.468796 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:07.501854 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:07.509958 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:07.513087 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:07.522294 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:07.522601 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:07.523115 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:07.527607 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:07.556666 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:07.557693 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:07.582202 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:07.591346 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:02:07.611577 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:07.613345 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:07.618794 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:07.623723 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:07.628453 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:07.629371 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:07.630999 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:02:07.637539 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:07.720355 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:07.723788 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:07.738668 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:07.748898 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:07.755708 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:07.764196 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:07.780330 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:07.784106 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:07.794613 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:07.804361 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:07.813664 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:07.814764 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:07.897678 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:07.899307 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:07.900758 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:07.903373 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:07.904004 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:07.931277 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:07.959913 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:07.961360 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:07.966984 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:07.968065 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:07.976709 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:07.980219 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:07.980502 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:07.995416 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:08.003626 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:08.027001 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:02:08.076009 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:08.094870 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:08.095799 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:08.099987 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:08.108474 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:08.110253 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:08.110837 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:08.118826 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 08:02:08.136158 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:08.188566 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:08.189615 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:08.191341 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:08.196281 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:08.200732 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:08.206174 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:08.283752 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:08.287635 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:08.288103 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:08.303428 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:08.307435 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:08.307705 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:08.308562 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:08.353288 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:08.353711 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:08.365113 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:08.372061 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:08.373603 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:08.376902 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:08.377012 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:08.392971 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:08.414788 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:08.425907 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:08.480875 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:08.483628 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:08.484431 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:08.488601 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:08.493491 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:08.496632 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:08.571506 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:08.574779 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:08.581876 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:08.589361 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:08.603596 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:08.605939 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:08.613694 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:08.615499 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:08.630293 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:08.631337 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:08.636799 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:08.643136 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:08.643224 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:08.654592 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:08.746304 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:08.753707 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:08.758793 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:08.760037 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:08.763540 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:08.763741 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:08.790915 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:08.807971 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:08.815912 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:08.819521 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:08.839066 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:08.846877 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:08.852701 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:08.916962 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:08.950381 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:08.950585 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:08.951540 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:08.957666 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:08.969436 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:08.973423 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:08.977026 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:08.993921 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:02:09.019690 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:09.022931 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:09.023021 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:09.025698 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:09.027290 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:09.030916 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:09.054695 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:09.115259 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240501 08:02:09.146865 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling FlushMRSOp(5e8550ea95984d08937b9b3e2af12bec): perf score=0.033387
I20240501 08:02:09.150041 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:09.153288 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:09.153486 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:09.168905 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:09.169652 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:09.180953 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:09.183123 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: FlushMRSOp(5e8550ea95984d08937b9b3e2af12bec) complete. Timing: real 0.036s user 0.009s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":560,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":195,"threads_started":2,"wal-append.queue_time_us":6900}
I20240501 08:02:09.184011 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling UndoDeltaBlockGCOp(5e8550ea95984d08937b9b3e2af12bec): 4839 bytes on disk
I20240501 08:02:09.184442 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: UndoDeltaBlockGCOp(5e8550ea95984d08937b9b3e2af12bec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240501 08:02:09.186252 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:09.192776 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:09.196607 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:09.207248 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:09.216174 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:09.222971 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:09.256554 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:09.317183 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:09.320272 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:09.332628 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:09.337330 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:09.346985 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:09.350062 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:09.398670 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:09.413997 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:09.417507 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:09.428006 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:09.437795 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:09.442238 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:09.450376 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:09.455170 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:09.456130 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:09.456789 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:09.458086 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:09.459827 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:09.477125 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:09.481089 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:02:09.498854 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:09.516047 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:09.564205 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:09.564196 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:09.568109 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:09.571964 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:09.597121 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:09.608244 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:09.660562 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:09.664788 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:09.665462 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:09.668649 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:09.670011 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:09.674121 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:09.693993 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:09.694088 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:09.716305 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:09.717137 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:09.717808 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:02:09.718329 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:09.776893 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:09.777990 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:09.780804 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:09.780911 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:09.788908 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:09.794013 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:09.802248 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling FlushMRSOp(ed88ea311f474ab28cc6fbe6069258bf): perf score=0.033370
I20240501 08:02:09.809070 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling FlushMRSOp(53193305f868433c852755e8a8a81766): perf score=0.033359
I20240501 08:02:09.842442 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:09.844050 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: FlushMRSOp(ed88ea311f474ab28cc6fbe6069258bf) complete. Timing: real 0.042s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1364,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":149,"threads_started":2,"wal-append.queue_time_us":336}
I20240501 08:02:09.844873 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling UndoDeltaBlockGCOp(ed88ea311f474ab28cc6fbe6069258bf): 4837 bytes on disk
I20240501 08:02:09.845387 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: UndoDeltaBlockGCOp(ed88ea311f474ab28cc6fbe6069258bf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240501 08:02:09.858934 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: FlushMRSOp(53193305f868433c852755e8a8a81766) complete. Timing: real 0.050s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":2610,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1528,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":498,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":148,"threads_started":2,"wal-append.queue_time_us":1331}
I20240501 08:02:09.859817 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling UndoDeltaBlockGCOp(53193305f868433c852755e8a8a81766): 4926 bytes on disk
I20240501 08:02:09.860291 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: UndoDeltaBlockGCOp(53193305f868433c852755e8a8a81766) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
W20240501 08:02:09.903899 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:09.921828 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:09.928376 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:09.928613 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:09.931087 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:09.934751 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:09.937595 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling FlushMRSOp(c95c4f3ef08f44d1b69fb7b98426f5af): perf score=0.033367
I20240501 08:02:09.942893 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:09.943415 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:09.946758 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:09.951640 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:09.955812 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:02:09.958977 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:09.959002 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:09.982591 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:09.983361 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:09.997431 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: FlushMRSOp(c95c4f3ef08f44d1b69fb7b98426f5af) complete. Timing: real 0.056s user 0.013s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":584,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":526,"threads_started":2,"wal-append.queue_time_us":2038}
I20240501 08:02:09.999116 13496 maintenance_manager.cc:419] P ca177138f2c44d2eafa1646b795f7608: Scheduling UndoDeltaBlockGCOp(c95c4f3ef08f44d1b69fb7b98426f5af): 4926 bytes on disk
I20240501 08:02:10.002966 13386 maintenance_manager.cc:643] P ca177138f2c44d2eafa1646b795f7608: UndoDeltaBlockGCOp(c95c4f3ef08f44d1b69fb7b98426f5af) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":139,"lbm_reads_lt_1ms":4}
I20240501 08:02:10.054042 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:10.058866 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:10.058964 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:10.061920 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:10.072865 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:10.073160 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:10.149017 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:02:10.164150 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:10.164544 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:10.167016 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:10.177202 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:02:10.178090 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:10.183878 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:10.188941 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling FlushMRSOp(bd8348990f1b496bb62d8c0484193442): perf score=0.033385
W20240501 08:02:10.195842 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:02:10.196063 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:10.199015 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:02:10.202808 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:02:10.209103 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: FlushMRSOp(bd8348990f1b496bb62d8c0484193442) complete. Timing: real 0.020s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":181,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1409,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":633,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":197,"threads_started":2,"wal-append.queue_time_us":190}
I20240501 08:02:10.209982 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling UndoDeltaBlockGCOp(bd8348990f1b496bb62d8c0484193442): 4583 bytes on disk
I20240501 08:02:10.210486 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: UndoDeltaBlockGCOp(bd8348990f1b496bb62d8c0484193442) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240501 08:02:10.328553 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:10.328729 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:10.338637 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:10.341607 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:10.349879 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:10.350775 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:10.352380 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:10.360893 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:10.366007 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:10.376929 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:10.395351 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:10.396456 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:10.397010 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:10.409188 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:02:10.411044 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:10.412760 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:10.420507 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:02:10.420563 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:10.425514 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:10.437012 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:10.453922 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:10.516419 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:10.555912 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:10.559864 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:10.560073 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:10.570986 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:10.573333 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:10.581192 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:10.637316 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:10.676327 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:10.678772 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:02:10.693182 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:10.694509 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:02:10.697669 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:10.703544 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:10.707741 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:10.714022 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:02:10.715106 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling FlushMRSOp(5e8550ea95984d08937b9b3e2af12bec): perf score=0.033362
I20240501 08:02:10.717589 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling FlushMRSOp(5b7de08d2f004cd8abc2bbe1cf7b1444): perf score=0.033376
I20240501 08:02:10.735997 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: FlushMRSOp(5b7de08d2f004cd8abc2bbe1cf7b1444) complete. Timing: real 0.018s user 0.014s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2526,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":669,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":162,"threads_started":2,"wal-append.queue_time_us":231}
I20240501 08:02:10.737113 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling UndoDeltaBlockGCOp(5b7de08d2f004cd8abc2bbe1cf7b1444): 4585 bytes on disk
I20240501 08:02:10.737596 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: UndoDeltaBlockGCOp(5b7de08d2f004cd8abc2bbe1cf7b1444) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240501 08:02:10.745959 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: FlushMRSOp(5e8550ea95984d08937b9b3e2af12bec) complete. Timing: real 0.029s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":168,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":995,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":539,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":176,"threads_started":2,"wal-append.queue_time_us":173}
I20240501 08:02:10.746932 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling UndoDeltaBlockGCOp(5e8550ea95984d08937b9b3e2af12bec): 4839 bytes on disk
I20240501 08:02:10.750578 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:10.751765 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: UndoDeltaBlockGCOp(5e8550ea95984d08937b9b3e2af12bec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 08:02:10.755380 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:10.788151 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:10.797955 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:10.803975 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:10.810659 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:10.846307 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:10.852500 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling FlushMRSOp(25cee7e51c7c438aa721d71fb82dc71e): perf score=0.033394
I20240501 08:02:10.854642 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:10.857715 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:10.872601 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:10.897143 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:10.898494 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:10.899473 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:10.902503 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:10.918663 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:10.920549 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:10.921662 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:10.922814 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:10.951979 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: FlushMRSOp(25cee7e51c7c438aa721d71fb82dc71e) complete. Timing: real 0.099s user 0.017s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":173,"dirs.run_cpu_time_us":250,"dirs.run_wall_time_us":21981,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":636,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":158,"threads_started":2,"wal-append.queue_time_us":50427}
I20240501 08:02:10.952946 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling UndoDeltaBlockGCOp(25cee7e51c7c438aa721d71fb82dc71e): 4836 bytes on disk
I20240501 08:02:10.953377 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: UndoDeltaBlockGCOp(25cee7e51c7c438aa721d71fb82dc71e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240501 08:02:10.975123 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:10.978509 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:10.981943 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:10.983346 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:10.984961 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:10.989488 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling FlushMRSOp(164fd265561e47128e7ff585b8cfab07): perf score=0.033362
I20240501 08:02:11.002847 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:11.025522 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: FlushMRSOp(164fd265561e47128e7ff585b8cfab07) complete. Timing: real 0.036s user 0.015s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1460,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11008,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":80,"threads_started":1,"wal-append.queue_time_us":174}
I20240501 08:02:11.026474 11579 maintenance_manager.cc:419] P 22ec12774a4047a8b859743dc73f8ff6: Scheduling UndoDeltaBlockGCOp(164fd265561e47128e7ff585b8cfab07): 4926 bytes on disk
I20240501 08:02:11.027803 11469 maintenance_manager.cc:643] P 22ec12774a4047a8b859743dc73f8ff6: UndoDeltaBlockGCOp(164fd265561e47128e7ff585b8cfab07) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":419,"lbm_reads_lt_1ms":4}
W20240501 08:02:11.071228 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:11.096978 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:11.100661 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:11.100795 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:11.104552 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:11.112982 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:02:11.115556 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:11.119292 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:11.177502 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:11.183807 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:11.193197 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:11.197094 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:11.207027 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:11.208155 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:11.286154 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:11.286653 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:11.292405 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:11.300881 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:11.304250 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:11.320040 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:11.331595 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:11.336999 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:11.350003 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:11.351155 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:11.355866 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:11.358583 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:11.360641 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:11.371618 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:11.419236 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:02:11.438464 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:11.444139 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:11.445820 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:11.452617 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:11.455161 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:11.462770 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:11.499995 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:11.533032 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:11.535914 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:11.539233 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:11.542277 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:11.550696 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:11.573510 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:11.631641 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 08:02:11.653530 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:11.666235 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:11.670372 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:11.680052 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:11.681942 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:11.696337 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:11.696524 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:02:11.717685 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:11.746384 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:11.749647 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:11.763492 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:11.768301 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:11.780086 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:11.789896 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:11.797434 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240501 08:02:11.801074 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:11.816861 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:11.817523 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:11.818167 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:11.830039 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:11.831063 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:11.879964 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:02:11.903476 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:11.906992 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:11.908030 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:11.914083 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:11.920827 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:11.923826 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:11.951068 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:11.973471 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:11.976779 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:11.985554 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:11.990592 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:11.991478 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:11.995550 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:02:11.997006 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:12.005928 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:02:12.108572 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:12.108654 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:12.111054 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:12.111251 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:12.116569 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:12.130813 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:12.183233 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:12.186218 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:12.188191 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:12.202482 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:02:12.205313 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:12.207365 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:12.212796 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:12.236109 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:12.256028 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:02:12.284289 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:12.284327 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:12.286553 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:12.288511 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:12.289475 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:12.292227 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:12.298193 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:12.311522 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling FlushMRSOp(5e8550ea95984d08937b9b3e2af12bec): perf score=0.033393
W20240501 08:02:12.327988 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:12.328213 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: FlushMRSOp(5e8550ea95984d08937b9b3e2af12bec) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":2896,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":2579,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2290,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":174,"threads_started":2,"wal-append.queue_time_us":341}
I20240501 08:02:12.329099 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling UndoDeltaBlockGCOp(5e8550ea95984d08937b9b3e2af12bec): 4839 bytes on disk
I20240501 08:02:12.329511 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: UndoDeltaBlockGCOp(5e8550ea95984d08937b9b3e2af12bec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
W20240501 08:02:12.339877 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:12.364313 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:12.379244 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:12.406617 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:02:12.407387 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:12.411945 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:02:12.418908 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:12.420897 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:12.423390 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:12.424803 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:12.442679 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:12.449287 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:12.504436 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:12.505183 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:12.522017 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:12.533869 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:12.555166 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:12.570237 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:12.581094 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:02:12.609541 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:12.611743 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:12.611892 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:12.613654 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:12.614492 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:12.615391 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:12.639626 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:12.679323 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:02:12.682337 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:12.682830 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:12.684916 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:12.698710 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:12.701885 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:12.709038 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:12.710758 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:12.733004 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:02:12.733778 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:12.787355 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:12.789276 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:12.793483 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:12.793561 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:12.797282 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:12.807884 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:12.845708 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:12.878576 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:12.883149 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:12.883682 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:12.884702 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:12.894986 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:12.922679 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:12.943710 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:12.947960 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:02:12.974303 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:12.975879 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:12.980216 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:12.983762 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:12.994817 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:12.998055 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:13.028052 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:02:13.046332 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:13.049948 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:13.051100 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:02:13.056267 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:13.060798 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:13.068951 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:13.084488 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:13.117466 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:02:13.125573 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:13.128859 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:13.135946 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:13.136247 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:13.142984 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:13.144620 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:13.200322 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:02:13.215116 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:13.218871 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:13.228536 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:02:13.234244 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:02:13.234935 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:13.240931 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:13.241813 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:13.242312 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:13.243335 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:13.262741 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:13.263499 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:13.288131 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:02:13.312528 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:13.318017 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:13.319397 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:13.322753 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:13.327915 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:13.354909 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:13.451566 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:13.456790 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:13.456809 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:13.464454 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:13.482023 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:13.496480 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:13.500614 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:13.500885 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:13.511910 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:13.516839 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:02:13.517452 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:13.527143 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:13.542546 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:13.594283 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:13.595885 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:13.600162 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:13.606501 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:13.611711 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:13.618664 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:13.702999 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:13.707477 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:13.710187 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:13.711506 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:13.740160 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:13.747445 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:13.781204 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:13.787590 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:13.789048 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:02:13.792256 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:13.792264 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:13.793694 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:13.798122 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:13.800318 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:13.804154 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:13.846499 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:02:13.910909 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:13.928193 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:13.931016 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:13.933547 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:13.935194 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:13.941231 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:13.944120 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:13.947067 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:13.955233 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:13.966780 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:13.968014 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:13.988572 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:14.110067 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:14.110571 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:14.121176 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:14.121899 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:14.123498 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:14.138931 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:02:14.141964 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:14.186093 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:14.189422 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:14.204749 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:14.249101 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:14.249166 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:14.250375 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:14.261272 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:14.266671 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:14.267500 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:14.280118 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:02:14.350159 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:14.361743 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:14.371488 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:14.373399 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:14.400425 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:14.404062 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:14.404448 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:14.404911 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:14.406096 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:14.408250 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:14.421221 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:14.447646 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:02:14.461814 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:14.509371 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:14.511055 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:14.511377 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:14.512210 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:14.529675 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:14.533052 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:14.552933 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:14.584834 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:02:14.602926 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:14.603199 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:14.605885 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:14.606561 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:14.608229 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:02:14.617993 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:14.618875 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:14.649569 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:14.682745 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:14.683771 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:14.692917 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:14.698869 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:14.700821 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:14.700935 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:14.701231 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:14.735805 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:14.761029 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:14.765594 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:02:14.803637 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:14.808498 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:14.808897 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:14.810576 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:14.853196 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:14.866647 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:14.869786 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:14.872263 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:14.884475 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:14.884739 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:14.885432 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:14.895457 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:14.936669 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:02:14.962208 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:14.962348 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:14.962689 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:14.971362 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:14.978876 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:14.989862 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:15.009029 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:15.020213 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:02:15.036569 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:02:15.046934 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:15.056727 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:15.058045 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:15.065941 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:15.069756 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:15.080101 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:02:15.081171 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:15.136191 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:15.137292 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:15.145503 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:15.150840 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:15.154852 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:15.155874 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:02:15.157325 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:15.174983 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:15.187559 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:15.245695 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:02:15.259373 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:15.264776 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:15.268347 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:15.268680 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:15.290724 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:15.292881 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:15.301985 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:15.303333 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:15.305846 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:15.306305 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:15.315687 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:15.326439 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:02:15.331041 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:15.346509 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:02:15.399528 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:15.399612 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:15.409512 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:15.409759 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:15.423002 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:15.434535 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:02:15.434635 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:15.456753 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:15.483384 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:02:15.487351 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:15.487771 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:15.488160 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:15.488667 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:15.492218 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:15.510171 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:15.580782 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:15.586241 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:15.588624 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:15.592590 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:15.599875 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:15.600893 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:15.621817 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:15.672461 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:15.675566 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:15.676033 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:15.681802 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:15.702180 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:15.702664 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:15.706701 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:15.762691 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:02:15.763648 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:15.767336 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:15.767683 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:15.768693 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:15.768944 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:15.769724 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:15.783812 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:15.896733 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:02:15.909940 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:15.912118 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:15.917925 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:15.920240 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:15.921725 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:15.928818 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:15.951121 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 08:02:15.964619 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:16.020977 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:02:16.045411 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:16.045980 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:16.047766 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:02:16.055980 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:16.064543 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:16.067433 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:16.074615 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:16.078033 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:16.081727 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:16.101891 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:16.109776 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:16.130049 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:16.134630 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:16.178467 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:16.179198 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:16.184764 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:16.187687 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:16.191521 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:16.194074 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:16.197850 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:16.246014 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:16.293602 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:16.298820 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:16.301091 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:16.313831 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:16.320576 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:16.328578 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:16.345955 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:16.349701 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:16.359694 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:16.362187 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:16.372133 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:16.372155 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:16.460016 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:02:16.475234 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:16.477937 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:16.483512 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:16.491796 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:16.497673 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:16.525789 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:16.548321 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:16.552572 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:16.555757 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:16.565980 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:16.577947 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:16.579952 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:16.593971 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:16.609241 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:16.613741 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:02:16.614261 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:16.617614 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:16.623082 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:16.627296 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:16.658341 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:02:16.673476 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:16.718971 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:16.740204 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:16.746970 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:16.773000 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:16.773672 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:16.774691 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:16.779160 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:16.788455 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:16.792140 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:16.798393 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:02:16.805138 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:16.824818 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:16.825135 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:16.837507 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:16.857479 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:16.935326 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:02:16.970880 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:16.974926 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:16.975131 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:16.980481 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:16.986802 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:17.002506 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:17.008064 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:17.009512 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:02:17.011211 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:17.014585 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:17.018107 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:17.027593 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:17.030879 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:17.034598 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:17.035131 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:02:17.041317 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:17.056943 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:17.060866 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:17.062990 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:02:17.064318 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:17.083165 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:17.086068 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:17.090031 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:17.121744 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:17.137817 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:17.146070 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:02:17.198375 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:17.204569 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:17.208261 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:17.209445 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:17.210611 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:17.215548 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:17.223961 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:17.225603 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:17.238727 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:17.240473 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:17.242002 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:17.247025 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:17.247556 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:17.392236 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:17.399335 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:17.404762 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:17.414899 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:17.415624 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:17.416798 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:17.418401 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:17.429505 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:17.434823 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:17.442914 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:17.450325 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 08:02:17.454214 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:17.478966 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:17.498505 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:02:17.502452 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:17.505908 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:02:17.507030 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:17.515508 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:17.525563 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:17.532179 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:17.534785 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:17.565191 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:17.597640 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:17.661154 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:17.667405 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:17.673465 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:17.688423 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:17.703733 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:17.821430 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:17.824342 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:17.824421 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:17.833659 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:17.838061 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:17.851531 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:17.853030 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:17.866405 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:17.872452 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:17.886505 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:02:17.887189 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:17.888010 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:17.898705 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:17.961316 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:17.964953 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:17.969802 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:17.970813 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:17.981891 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:02:17.989181 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:17.991194 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:18.010006 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:02:18.059782 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:18.062937 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:18.069109 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:18.071156 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:18.073459 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:18.079819 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:02:18.099879 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:18.100826 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:18.103741 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:18.105047 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:18.105402 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:02:18.107224 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:18.124065 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:18.127982 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:18.154003 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:18.196990 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:18.202900 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:02:18.251543 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:18.251664 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:18.260394 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:18.265208 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:18.267151 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:18.268044 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:18.285635 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:02:18.290323 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:18.302779 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:02:18.304488 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:18.306354 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:18.312902 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:18.320605 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:18.337919 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:18.346642 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:18.387601 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:02:18.428918 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:18.429678 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:18.435858 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:18.447523 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:18.448869 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:18.472792 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:18.514878 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:18.516181 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:18.517156 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:18.533823 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:18.535275 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:18.539744 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:18.545204 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:18.549556 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:18.572422 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:18.572691 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:18.573580 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:18.573807 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:18.588303 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:18.589651 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:02:18.701905 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:18.702494 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:18.703933 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:18.703950 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:18.706240 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:18.713721 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:18.723166 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:18.728258 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:18.747156 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:18.758145 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:18.760370 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:18.771224 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:18.776769 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:18.788442 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:18.853268 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:02:18.896006 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:18.900494 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:18.901608 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:18.913110 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:18.914268 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:18.914798 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:18.916450 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:18.919633 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:18.927414 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:18.930526 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:18.931206 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:18.948463 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:02:18.961107 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:19.019755 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:19.024331 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:19.030373 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:19.043076 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:19.046028 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:19.048511 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:19.103628 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:19.111658 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:19.113128 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:19.114849 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:19.123209 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:19.147078 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:19.193428 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:19.195207 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:19.195669 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:19.196239 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:19.198523 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:19.221067 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:19.227676 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:19.256052 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:02:19.307303 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:19.309561 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:19.317502 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:19.319579 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:19.340376 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:19.340868 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:19.346581 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:19.369633 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:19.373356 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:19.377925 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:19.381834 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:19.385776 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:02:19.387415 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:19.389431 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:19.406034 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:19.437351 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:19.454991 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:02:19.481065 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:19.486888 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:19.488301 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:19.488804 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:02:19.492658 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:19.496590 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:19.513854 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:19.575798 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:02:19.594717 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:19.599264 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:19.600744 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:19.601218 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:19.602607 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:19.617475 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:19.648816 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:19.660637 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:02:19.756419 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:02:19.762981 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:19.775141 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:19.779891 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:19.783361 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:19.793834 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:19.799605 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:19.800796 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:19.807255 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:19.809298 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:19.811029 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:19.818593 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:19.825616 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:19.826859 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:19.834002 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:19.842182 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:19.842468 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:19.863479 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:19.875257 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:19.882867 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:19.897243 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:19.931648 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:02:20.023712 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:20.024456 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:20.033349 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:20.039036 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:20.044123 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:20.048477 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:02:20.054917 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:20.062628 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:20.065392 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:20.068338 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:20.082531 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:20.084048 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:20.085307 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:02:20.106498 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:20.187633 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:20.211964 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:20.212504 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:20.212949 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:20.217418 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:20.224232 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:02:20.226086 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:20.236129 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:02:20.243642 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:20.248670 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:20.255337 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:20.257256 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:20.262267 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:20.262699 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:20.269007 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:20.275372 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:20.283294 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:20.296033 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:02:20.321349 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:20.322814 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:20.328677 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:20.329525 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:20.333262 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:20.342394 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:20.370754 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:20.400645 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:20.456467 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:02:20.456859 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:20.460093 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:20.461530 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:20.464046 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:20.467512 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:20.476267 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:20.486814 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:02:20.496191 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:20.498659 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:20.501253 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:20.502750 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:20.514993 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:20.517297 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:20.523146 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:20.602176 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:20.605379 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:20.650765 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:20.661919 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:20.661980 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:20.662012 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:20.672840 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:20.704131 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:20.710700 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:20.713898 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:20.716830 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:20.725955 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:02:20.733484 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:20.739580 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:20.750965 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:20.751897 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:20.758867 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:20.761678 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:20.766389 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:20.771557 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:20.783466 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:20.820094 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:20.839570 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:20.849823 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:02:20.912628 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:20.912773 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:20.915413 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:20.918637 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:20.921350 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:20.928181 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:21.004508 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:02:21.024760 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:21.027215 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:21.029806 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:21.031235 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:21.036259 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:21.050591 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:02:21.059826 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:21.090687 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:02:21.151683 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:21.151974 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:21.152283 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:21.152796 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:21.156322 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:21.162645 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:21.180114 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:21.184049 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:21.188935 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:21.195067 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:21.198356 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:21.221729 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:21.225978 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:21.289959 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:21.298689 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:21.299643 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:21.300170 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:21.302141 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:21.319936 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:21.338660 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:21.374562 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:21.376919 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:21.382872 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:21.384922 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:21.385190 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:21.401891 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:21.415457 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:21.488353 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:02:21.499752 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:21.501174 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:21.502389 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:21.508460 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:21.514562 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:21.517825 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:21.583473 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:21.587484 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:21.588151 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:21.589618 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:21.590011 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:21.616516 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:02:21.618983 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:21.679353 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:21.684376 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:21.684494 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:21.687415 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:21.688304 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:21.693753 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:21.702996 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:21.755179 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:21.758585 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:02:21.759289 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:02:21.771548 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:21.772848 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:21.774076 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:21.780565 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:21.782377 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:21.847132 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:02:21.888682 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:21.892601 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:21.909791 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:21.927397 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:21.927778 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:21.937325 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:21.948552 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:21.948902 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:21.949244 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:21.953795 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240501 08:02:21.955865 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:21.956053 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:21.956156 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:21.965230 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:21.965337 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:21.967530 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:02:22.011687 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:22.013243 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:22.019775 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:22.028090 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:22.029109 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:22.031010 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:22.124459 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:22.125144 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:22.125449 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:22.128325 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:22.133901 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:22.142956 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:22.184959 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:22.199699 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:22.201757 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:22.204182 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:22.205755 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:22.206090 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:22.212219 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:22.310647 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:22.315780 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:22.321012 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:02:22.322816 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:22.325896 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:22.326051 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:22.326701 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:22.338763 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:22.358747 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:22.366438 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:22.366604 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:22.366920 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:22.378186 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:22.385696 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:22.390635 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:22.424855 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:22.444183 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:22.451542 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:22.453249 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 08:02:22.461937 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:22.464406 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:22.471957 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:22.480415 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:22.487062 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:22.502132 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:22.516511 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:22.551383 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:22.558301 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:22.558571 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:22.559501 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:22.568058 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:22.568610 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:22.671180 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:02:22.685475 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:22.685932 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:22.687964 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:22.690491 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:22.694756 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:22.699906 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:02:22.703824 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:22.708818 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:22.712145 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:22.716496 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:22.729897 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:22.772869 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:02:22.775923 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:02:22.776273 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:22.780055 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:22.785050 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:02:22.792001 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:22.792773 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:22.801195 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:22.803668 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:02:22.813587 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:22.829416 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:22.849296 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:22.855065 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:22.857621 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:22.873217 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:22.877409 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:22.889323 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:22.948611 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:22.953795 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:22.956004 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:22.962111 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:02:22.965306 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:22.979856 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:22.981196 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:23.021476 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:23.026640 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:23.026783 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:23.031047 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:23.035142 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:23.037335 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:23.137030 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:23.141062 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:23.151206 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:23.151947 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:23.155027 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:23.160190 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:23.164176 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:23.170542 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:23.171635 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:02:23.178283 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:23.194614 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:23.195127 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:23.198544 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:23.205849 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:23.220557 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:23.301663 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:23.302590 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:23.313411 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:23.313918 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:23.315008 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:23.322618 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:23.327323 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 37 similar messages]
I20240501 08:02:23.394026 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:23.398730 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:23.400223 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:23.403008 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:23.405103 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:23.418081 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:23.493629 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:02:23.525311 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:23.525800 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:23.526207 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:23.527021 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:02:23.538599 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:23.538937 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:23.542526 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:23.557881 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:23.567098 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:23.574156 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:23.579428 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:23.585009 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:23.595646 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:23.604928 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:23.605244 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:23.655184 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:02:23.700598 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:23.703497 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:23.707487 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:23.707736 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:23.731896 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:23.744693 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:23.762642 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:23.764267 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:23.764549 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:23.770265 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:23.777352 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:23.780951 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:23.813200 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:23.819759 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:02:23.854228 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:23.857098 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:23.864125 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:23.867035 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:23.868373 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:23.877851 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:23.970692 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:23.977509 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:23.978639 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:23.989084 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:23.990557 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:24.018026 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:24.019178 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:24.026249 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:24.030685 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:24.035629 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:24.041170 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:24.061408 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:02:24.066079 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:24.066813 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:24.080842 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:24.125193 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:02:24.143129 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:24.151428 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:24.153776 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:24.153972 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:24.160425 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:24.173451 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:24.199455 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:24.202769 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:24.211895 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:24.213778 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:24.215616 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:24.220103 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:24.230170 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:24.249245 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:02:24.323438 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:24.323427 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:24.323943 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:24.324230 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:24.336055 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:24.336356 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:24.429926 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:24.440899 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:24.442636 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:24.447561 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:24.460937 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:24.463557 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:24.465094 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:02:24.482766 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:24.482998 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:24.488576 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:24.489660 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:24.503381 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:24.514106 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:24.540771 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:24.616317 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:02:24.617741 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:02:24.621030 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:24.626888 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:24.634297 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:24.634384 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:24.638239 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:24.658180 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:24.658226 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:24.663878 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:02:24.678048 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:24.684705 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:24.688433 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:24.692659 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:24.692862 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:24.703310 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:24.709595 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:24.740670 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:02:24.774840 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:24.777917 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:24.782086 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:24.784128 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:24.791224 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:24.805497 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:24.811951 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:02:24.878090 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:24.881425 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:24.881691 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:24.881693 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:24.881884 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:24.884254 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:24.964118 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:24.966199 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:24.968140 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:24.970263 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:24.974215 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:24.987814 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:24.993899 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:25.000566 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:25.011725 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:25.025738 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:02:25.042130 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:25.042706 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:25.042814 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:25.046000 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:25.066272 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:25.067864 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:25.078461 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:25.103926 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:25.105669 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:25.106460 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:25.108476 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:25.115756 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:25.117189 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:25.121325 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:25.200276 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:25.219712 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:02:25.229885 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:25.233758 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:25.236861 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:02:25.237967 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:25.238701 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:25.239079 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:25.248706 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:25.290477 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:02:25.299997 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:02:25.333489 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:25.333949 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:25.346630 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:25.348423 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:25.366855 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:25.375334 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:25.413445 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:25.431792 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:02:25.432567 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:25.433401 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:02:25.446837 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:25.448706 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:25.461539 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:25.489778 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:25.491216 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:25.493742 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:25.496623 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:25.506613 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:25.515244 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:25.520910 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:25.543709 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:25.557982 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:25.566154 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:25.571774 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:25.577245 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:25.584920 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:25.591820 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:25.616190 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:25.632457 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:25.671682 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:02:25.702311 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:25.708160 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:25.715348 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:25.717123 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:25.725605 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:25.726573 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:25.739609 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:02:25.785581 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:25.786592 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:25.787027 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:25.787142 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:25.795430 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:25.811041 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:25.920094 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:25.920711 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:02:25.921087 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:25.923961 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:25.933949 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:25.937331 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:25.937600 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:25.963255 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:25.966076 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:25.966213 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:25.970917 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:25.974927 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:25.996799 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:26.085119 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:26.085604 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:26.090155 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:26.094054 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:26.094496 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:26.096274 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:26.126835 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:26.157586 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:26.163985 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:02:26.165153 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:26.165277 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:26.180317 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:26.180630 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:26.195807 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:26.198563 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:26.198590 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:26.220168 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:02:26.239488 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:02:26.273942 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:26.277225 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:26.287526 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:26.291193 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:26.301221 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:26.304308 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:02:26.314251 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:26.376251 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:26.376806 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:26.376916 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:26.390075 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:26.390753 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:26.397711 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:26.399165 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:26.413048 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:26.418597 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:26.421423 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:26.427182 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:26.439325 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:26.526849 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:26.527451 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:26.541349 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:26.550561 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:26.550577 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:26.559249 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:26.611713 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:26.614746 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:26.617630 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:26.628113 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:26.628989 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:26.648538 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:26.657780 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:26.705188 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:26.715335 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:26.719657 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:26.726054 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:26.726104 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:26.728315 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:26.729081 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:26.737747 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:26.759266 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:26.797236 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:26.809623 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 83 similar messages]
I20240501 08:02:26.814188 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:26.824121 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:26.839037 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:26.840873 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:26.850167 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:26.864504 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:26.875798 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:02:26.876646 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:26.884071 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:26.889945 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:26.897939 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:26.900656 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:26.908159 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:26.951426 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:26.956507 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:02:26.967850 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:26.970265 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:26.971948 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:26.972793 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:26.977442 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:26.986780 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:26.993404 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:02:27.066686 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:02:27.079528 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:27.084906 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:27.085273 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:27.091713 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:27.093756 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:27.100524 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:27.110289 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:27.175760 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:27.177819 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:27.212078 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:27.212301 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:27.221697 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:27.234567 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:27.241803 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:27.277874 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:27.278151 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:27.278595 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:27.280642 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:27.296065 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:27.298058 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:27.299983 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:27.303216 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:27.317477 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:27.317763 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:27.320591 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:27.321568 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:27.321710 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:27.329890 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:27.412113 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:27.466410 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:02:27.467334 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:27.468534 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:27.471551 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:27.474613 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:27.477972 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:27.479809 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:27.481355 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:27.487780 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:02:27.496735 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:27.574067 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:02:27.603945 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:27.607842 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:27.618640 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:27.620131 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:27.630630 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:27.639156 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:27.639961 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:27.643457 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:27.714907 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:27.719254 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:27.739221 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:27.740510 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:27.745246 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:02:27.753378 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:27.766103 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:02:27.780730 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:27.785609 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:27.789830 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:27.795558 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:27.805037 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:27.817761 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:27.820541 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:27.832643 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:27.835311 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:02:27.844430 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:27.844477 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:27.852983 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:27.863552 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:27.870985 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:02:27.871114 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:27.895850 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:27.921722 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:27.963593 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:02:27.965420 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:27.969784 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:27.971365 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:27.974264 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:27.993901 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:28.013160 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:28.039701 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:28.042618 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:02:28.045276 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:28.048076 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:28.057808 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:28.057833 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:28.059832 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:28.073563 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:28.074587 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:02:28.162779 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:28.181805 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:28.188711 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:28.209043 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:28.209022 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:28.213130 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:28.217141 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:28.219722 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:28.220878 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:28.223620 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:28.231426 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:28.231865 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:28.243650 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:28.282601 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:28.284019 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:28.289031 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:28.290585 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:28.304831 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:28.314980 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:28.327487 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:28.417258 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 08:02:28.432313 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:28.434037 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:28.435516 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:28.439553 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:28.448448 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:28.451503 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:28.461311 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:28.462531 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:28.464378 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:28.485576 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:28.496536 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:28.495599 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:28.522037 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:02:28.615669 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:28.624876 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:28.626528 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:28.630813 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:28.633292 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:28.644351 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:28.646268 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:28.682611 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:28.687753 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:28.704259 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:28.711870 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:28.713426 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:28.741616 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:02:28.753549 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:28.763782 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:28.768743 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:28.768743 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:28.777737 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:28.778604 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:28.780400 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:28.802498 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:28.802654 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:28.804176 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:28.870810 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:28.876084 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:28.876722 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:28.877127 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:28.879045 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:28.899310 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:28.988423 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:28.991979 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:28.995208 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:29.001093 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:29.018064 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:29.035506 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:29.065162 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:29.066253 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:29.070750 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:29.073239 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:29.078531 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:29.107936 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:29.114598 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:29.176331 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:29.184017 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:29.185550 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:29.190943 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:29.192831 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:29.193339 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:29.201151 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:29.202658 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:29.203341 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:29.204856 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:02:29.213603 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:29.223512 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:29.233359 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:29.237131 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:29.248708 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:29.291289 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:29.318917 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:02:29.349220 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:29.353046 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:29.359380 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:29.361680 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:29.362318 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:29.369870 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:29.386631 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:29.461742 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:29.473179 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:29.476696 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:29.481253 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:29.482602 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:29.495281 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:02:29.507583 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:29.514550 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:29.531527 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:29.534086 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:29.538914 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:29.549535 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:29.563230 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:29.571128 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:29.593102 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:29.629438 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:29.631878 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:29.645610 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:29.650616 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:29.667749 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:29.682054 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:29.701335 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:29.715119 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:29.718300 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:29.725625 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:29.727429 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:29.752658 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:02:29.760485 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:29.770294 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:29.773875 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:29.779438 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:29.789703 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:29.797977 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:29.799327 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:29.802825 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:29.805580 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:29.828387 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:29.850692 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:29.893981 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:02:29.921347 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:29.926352 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:29.927668 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:29.935138 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:29.940656 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:29.942646 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:29.950862 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:29.953310 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:29.968591 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:29.973583 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:29.975692 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:29.995014 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:29.997367 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:29.998569 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:29.998874 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:30.010702 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:30.088630 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:30.120566 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:30.123152 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:30.134399 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:30.144820 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:30.152729 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:30.160550 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:30.167003 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:30.167271 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:30.183279 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:30.183537 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:30.201983 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:30.206826 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:30.209518 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:30.217381 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:30.253513 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:30.256629 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:30.262390 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:30.264995 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:30.278952 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:30.288556 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:30.310907 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:30.348131 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:30.350376 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:30.359359 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:30.360926 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:30.368103 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:30.369562 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:30.374650 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:30.385306 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:30.424897 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:30.444242 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:30.445848 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:30.451550 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:30.453084 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:30.455474 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:30.459914 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:30.508148 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:30.519585 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:30.523726 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:30.529039 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:30.539522 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:30.542502 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:30.543720 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:30.551743 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 08:02:30.563704 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:30.597721 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:30.598490 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:30.603377 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:30.607767 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:30.616293 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:30.618091 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:30.644614 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:30.714880 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:30.715360 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:30.715792 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:30.719942 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:30.720120 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:30.730823 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:30.780654 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:30.815402 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:30.816447 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:30.817206 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:30.817761 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:30.820248 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:30.843169 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:30.846479 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:30.874403 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:30.929745 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:30.934065 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:30.939373 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:30.948431 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:30.956234 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:30.957589 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:30.996680 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:31.000104 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:31.002312 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:31.006604 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:31.016302 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:31.024191 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:31.042768 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:31.071996 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:02:31.090256 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:31.093024 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:31.101446 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:31.102753 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:31.111946 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:31.115545 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:31.122848 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:31.187990 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:31.190804 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:31.190881 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:31.195199 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:31.204459 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:31.205483 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:31.274713 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:31.277299 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:31.279445 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:02:31.279611 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:31.291761 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:31.292793 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:31.292997 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:31.305774 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:31.384239 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:31.388626 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:31.388676 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:31.389583 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:31.389616 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:31.398572 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:31.409854 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:31.415478 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:31.421989 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:31.425071 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:31.427086 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:31.432308 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:31.432912 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:31.435453 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:31.435513 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:31.526834 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:31.530663 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:31.536062 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:31.536448 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:31.544540 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:31.563604 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:31.592521 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:31.602815 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:31.616225 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:31.623380 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:31.625348 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:31.633409 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:31.642304 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:31.645203 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:31.648437 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:31.721737 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:31.725241 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:31.727532 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:31.735023 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:31.755115 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:31.777760 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:31.831786 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:31.837671 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:31.839462 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:31.850654 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:31.854765 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:31.858676 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:31.863129 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:31.869426 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:31.872280 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:31.872967 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:31.881063 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:31.888309 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:31.894613 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240501 08:02:31.908854 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:31.911016 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:31.932165 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:31.941205 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:31.948408 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:31.982679 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:31.986501 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:31.993613 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:31.994196 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:31.997090 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:31.999188 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:32.006194 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:32.048779 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:32.054322 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:32.058468 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:32.064849 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:32.066498 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:32.071635 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:32.075372 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:02:32.107187 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:32.176126 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:32.181178 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:32.183012 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:32.183287 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:32.193435 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:32.214002 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:32.216954 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:32.242404 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:32.243638 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:32.258580 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:32.273355 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:32.280539 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:32.283269 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:32.285171 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:32.285280 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:32.301908 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:32.304754 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:32.308099 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:32.308578 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:32.308841 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:32.319767 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:32.344642 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:32.371084 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:32.413589 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:32.417897 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:32.419502 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:32.419872 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:32.420214 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:32.420929 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:32.443013 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:32.479568 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 08:02:32.480361 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:32.502941 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:32.581146 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:32.624495 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:32.651217 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:32.652808 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:32.661478 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:32.661530 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:32.671005 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:32.676681 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:32.715000 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:32.725199 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:32.735245 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:32.745435 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:32.747025 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:32.766368 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:32.767093 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:32.795742 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:32.838459 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:32.840196 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:32.861304 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:32.864099 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:32.870996 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:32.876578 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:32.908703 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:32.914374 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:32.916163 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:32.919209 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:32.929052 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:32.933966 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:32.935286 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:32.938359 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:32.993614 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:32.998036 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:32.998317 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:33.003275 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:33.004566 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:33.005355 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:33.015041 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:33.026139 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:33.032297 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:33.110973 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:33.117089 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:33.128926 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:33.130496 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:33.130796 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:33.144042 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:33.146639 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:33.173329 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:33.179582 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:33.182547 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:33.183151 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:33.189075 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:33.191470 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:33.197453 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:33.275110 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:02:33.276155 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:33.282078 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:33.289247 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:33.293855 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:33.301048 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:33.306141 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:33.310232 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:33.333727 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:33.335131 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:33.342063 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:33.345185 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:33.345649 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:33.412034 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:33.415663 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:33.447520 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:33.471357 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:33.482621 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:33.491304 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:33.500758 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:33.508984 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:33.510210 11598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 08:02:33.517145 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:33.521524 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:33.525749 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:33.541561 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:33.547488 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:33.550107 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:33.553622 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:33.558926 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:33.563621 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:02:33.576704 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:33.641326 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:33.641774 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:33.644407 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:33.646361 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:33.660531 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:33.662314 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:33.672317 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:33.757431 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:33.766515 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:33.778863 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:33.785056 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:33.787434 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:33.789842 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:33.791316 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:33.799690 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:33.818084 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:33.822166 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:33.822615 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:33.839000 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:33.861502 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:33.932826 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:33.950824 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:33.963493 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:33.966737 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:33.976477 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:33.981760 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:33.991581 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:34.000375 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:34.023597 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:34.026507 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:34.031708 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:34.043731 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:34.045100 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:34.053217 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:34.072465 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:34.086827 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:34.089673 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:34.101163 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:34.104100 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:34.112161 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:34.127694 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:34.160635 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:34.212397 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:34.212628 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:34.213006 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:34.222668 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:34.224439 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:34.237913 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:34.288455 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:34.288508 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:34.291388 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:34.292025 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:34.311074 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:34.311404 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:34.345424 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:34.410588 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:34.442488 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:34.457989 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:34.458158 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:34.459893 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:34.464702 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:34.475981 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:34.490573 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:34.495054 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:34.512499 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:34.558450 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:34.580252 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:34.582100 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:34.586115 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:34.589964 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:34.591085 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:34.598863 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:34.615104 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:34.669451 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:34.674006 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:34.675899 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:34.682296 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:34.689232 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:34.691151 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:34.710767 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:34.755281 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:34.765332 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:34.788398 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:34.790336 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:34.791082 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:34.795010 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:34.802826 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:34.808872 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:34.809880 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:34.835604 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:34.872452 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:34.877190 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:34.879108 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:34.885023 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:34.891148 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:34.894068 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:34.915213 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:02:34.928483 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:34.951370 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:34.953513 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:34.954219 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:34.966238 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:34.966936 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:34.967130 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:34.976275 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:35.031643 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:35.034296 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:35.035795 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:35.037827 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:35.044029 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:35.045161 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:35.047304 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:35.047646 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:35.097590 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:35.141034 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:35.141304 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:35.156745 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:35.159343 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:35.193867 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:35.199898 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:35.210397 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:35.210870 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:35.210948 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:35.214185 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:35.226063 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:35.238344 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:35.242360 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:35.302388 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:35.307454 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:35.345463 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:35.349361 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:35.351150 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:35.354914 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:35.360189 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:35.360464 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:35.363571 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:35.373258 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:35.382061 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:35.383103 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:35.420882 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:35.439422 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:35.442409 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:35.443269 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:35.445263 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:35.455574 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:35.461030 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:35.467965 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:35.474529 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:35.481098 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:35.486635 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:35.488605 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:35.501596 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:35.618755 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:35.620555 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:35.629976 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:35.643258 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:35.644858 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:35.646891 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:35.647008 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:35.650583 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:35.661353 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:35.677809 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:35.679984 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:35.698186 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:35.754029 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:35.808738 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:35.818475 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:35.820210 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:35.826658 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:02:35.830319 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:35.830905 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:35.832247 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:35.862231 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:35.866869 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:35.873426 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:35.891795 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:35.917627 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:35.920184 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:35.920660 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:35.922061 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:35.926159 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:35.945894 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:35.949563 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:35.953454 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:35.954708 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:35.955921 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:35.974635 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:35.982097 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:36.060788 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:36.078413 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:36.081683 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:36.082361 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:36.085640 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:36.095489 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:36.095489 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:36.135831 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:36.136384 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:36.140506 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:36.146718 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:36.151813 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:36.166894 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:36.168543 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:36.168929 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:36.198951 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:36.240698 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:02:36.252049 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:36.252373 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:36.255230 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:36.264991 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:36.265791 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:36.277117 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:36.288429 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:36.422834 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:36.432950 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:02:36.477814 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:36.479691 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:36.484684 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:36.490646 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:36.496009 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:36.527161 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:36.541049 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:36.549624 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:36.551453 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:36.556308 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:36.566895 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:36.578636 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:36.653124 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:36.657469 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:36.668936 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:36.672070 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:36.687376 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:36.691167 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:36.759169 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:36.770576 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:36.775550 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:36.776768 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:36.786024 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:36.787794 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:36.789660 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:36.790097 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:36.796463 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:36.799515 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:36.805053 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:36.806890 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:36.807273 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:36.842101 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:36.855525 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:36.899890 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:36.921008 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:36.921239 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:02:36.930153 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:36.964918 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:36.965260 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:36.973486 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:36.973582 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:36.983562 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:36.984367 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:36.985090 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:36.985467 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:36.989260 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:37.023979 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:37.026139 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:37.030006 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:37.035629 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:37.047946 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:37.051682 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:37.128873 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:37.162460 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:37.166291 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:37.170249 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:37.180826 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:37.184934 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:37.185266 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:02:37.188069 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:37.188261 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:37.194633 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:37.197695 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:37.206686 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:37.210611 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:37.231986 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:37.241022 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:37.241077 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:37.256765 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:37.263322 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:37.277694 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:37.278349 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:37.291714 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:37.294601 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:37.299825 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:37.307343 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:37.394321 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:37.410261 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:37.413246 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:37.415238 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:37.431103 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:37.452308 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:37.455916 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:37.462544 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:37.463541 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:37.465257 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:37.465495 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:37.480176 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:37.480546 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:37.491590 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:37.515393 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:02:37.604759 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:37.605410 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:37.606216 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:37.611618 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:37.612664 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:37.623565 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:37.639496 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:37.668442 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:37.671247 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:37.682845 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:37.689702 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:37.690223 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:37.695732 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:37.700377 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:37.700852 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:37.720465 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:37.723012 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:37.740116 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:37.740391 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:37.741065 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:37.742851 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:37.743909 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:37.769886 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:02:37.772538 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:02:37.784029 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:37.790634 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:37.896116 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:37.905846 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:37.906595 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:37.908416 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:37.909265 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:37.918200 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:37.921983 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:37.930138 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:37.947386 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:37.959892 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:37.961710 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:37.964010 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:37.983449 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:38.078151 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:38.083129 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:38.084640 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:38.086030 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:38.087488 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:38.089053 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:38.092238 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:38.093536 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:38.099185 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:38.110730 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:38.119136 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:38.124941 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:38.148995 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:38.151633 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:38.162074 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:38.248867 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:38.293911 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:38.295790 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:38.296339 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:38.296344 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:38.297878 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:38.299584 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:02:38.311035 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:38.369164 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:38.426803 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:38.431748 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:38.433454 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:38.435590 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:38.436218 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:38.438695 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:38.473181 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:38.514964 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 08:02:38.515656 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:38.520677 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:38.523628 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:38.529495 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:38.534914 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:38.535439 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:38.545840 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:38.570603 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:38.577443 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:02:38.597672 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:38.602541 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:38.608096 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:38.612478 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:38.619850 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:38.637295 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:38.670998 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:02:38.690822 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:38.694065 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:38.700992 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:38.702494 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:38.709816 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:38.716985 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:38.733072 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:38.771744 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:38.774735 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:38.778220 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:38.783363 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:38.787209 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:38.801865 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:38.872884 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:38.911859 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:38.915105 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:38.919217 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:38.939841 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:38.948117 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:38.957602 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:38.958465 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:38.964027 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:38.977006 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:38.983559 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:38.989568 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:39.006983 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:39.013345 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:39.015616 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:39.042686 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:39.047116 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:39.049036 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:39.049371 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:39.059415 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:39.077136 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:39.078665 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:39.171232 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:39.178903 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:39.183230 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:39.183429 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:39.186013 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:39.208077 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:39.232014 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:39.232079 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:39.234999 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:39.241657 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:39.244171 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:39.268776 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:39.315660 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:02:39.381392 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:39.387344 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:39.388257 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:39.388935 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:39.395789 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:39.399014 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:39.400207 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:39.416406 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:02:39.430971 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:39.432035 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:39.439486 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:39.445560 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:39.447460 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:39.451788 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:39.469277 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:39.470503 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:39.481007 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:39.523392 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:39.525575 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:39.530622 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:39.536592 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:39.540464 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:39.547688 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:39.580783 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:39.593716 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:39.603515 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:39.627230 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:39.630461 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:39.634569 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:39.639313 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:39.661938 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:39.661935 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:39.671787 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:39.674674 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:02:39.680208 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:39.680223 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:39.689707 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:39.699324 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:02:39.706068 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:39.707008 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:39.814067 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:39.814332 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:39.814834 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:39.817302 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:39.818504 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:39.828410 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:39.828866 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:39.937718 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:39.937882 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:39.939070 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:02:39.946318 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:39.958391 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:39.958678 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:39.960285 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:39.962463 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:39.969403 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:39.977145 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:39.982885 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:02:39.992074 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:39.993373 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:40.008647 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:40.011889 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:40.023051 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:40.108314 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:40.108531 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:40.109401 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:40.109833 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:40.112792 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:40.115273 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:40.129848 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:40.134243 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:40.148772 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:02:40.230049 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:40.230716 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:40.231057 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:40.233171 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:40.237167 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:40.237865 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:40.296558 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:40.372745 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:40.388200 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:40.389756 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:40.400177 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:40.400166 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:40.404789 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:40.408337 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:40.409822 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:40.415011 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:40.421120 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:40.444661 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:40.481170 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:40.484711 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:40.485108 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:02:40.488253 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:40.494587 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:40.496188 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:40.499256 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:40.502234 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:40.518007 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:40.545589 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:40.589102 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:02:40.619169 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:40.619181 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:40.627523 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:40.630861 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:40.631439 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:40.632678 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:02:40.633934 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:40.645159 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:40.683589 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:40.689046 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:40.694761 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:40.701817 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:40.703548 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:40.709225 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:40.746387 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:40.775456 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:02:40.859637 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:40.862563 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:40.867532 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:40.869206 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:40.874761 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:40.880425 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:40.886492 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:40.892885 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:40.894624 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:40.894809 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:40.899003 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:40.909096 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:02:40.919935 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:40.925412 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:40.925424 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:40.928314 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:40.955633 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:40.976697 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:40.977031 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:40.980545 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:40.992396 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:41.063513 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:02:41.090150 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:41.094293 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:41.107585 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:02:41.110020 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:41.110772 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:41.113543 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:41.115742 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:41.117575 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:41.123767 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:41.127137 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:41.129007 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:41.131937 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:02:41.140597 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:41.159754 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:41.164882 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:41.249269 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:02:41.303735 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:41.306808 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:41.307456 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:41.315033 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:41.319881 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:41.320775 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:41.324909 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:41.325376 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:41.325846 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:41.375845 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:41.390466 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:41.393043 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:41.395742 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:41.417181 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:41.423359 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:41.437841 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:41.451790 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:41.456418 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:41.468448 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:41.472043 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:41.518185 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:41.518891 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:41.520370 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:02:41.520967 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:41.522909 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:41.524528 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:41.531242 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:41.641211 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:41.642575 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:41.645625 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:41.649009 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:41.649034 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:41.675269 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:41.676903 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:41.696260 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:02:41.810838 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:41.814926 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:41.818960 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:41.829972 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:02:41.832053 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:41.847038 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:41.847654 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:41.868465 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:41.871724 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:41.875769 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:41.880260 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:41.881474 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:41.914984 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:41.918367 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:41.919708 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:41.921865 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:41.930780 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:41.934576 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:02:41.938572 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:41.948199 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:41.956393 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:41.988272 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:02:41.989084 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:42.039283 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:42.053822 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:42.054520 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:42.056648 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:42.058635 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:42.062860 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:42.069381 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:42.102476 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:42.105098 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:42.105785 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:42.107234 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:42.187079 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:42.187582 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:42.191743 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:42.196614 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:42.209049 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:42.210960 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240501 08:02:42.217438 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:42.248522 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:02:42.275386 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:42.278682 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:42.282294 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:42.282603 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:42.285811 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:42.287748 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:42.314671 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:42.353756 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:42.367209 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:42.370992 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:42.380463 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:42.381532 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:42.383790 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:42.451251 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:42.455907 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:42.462656 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:42.465458 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:42.466082 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:42.484174 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:42.527654 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 77 similar messages]
I20240501 08:02:42.532786 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:42.539853 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:42.546597 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:42.551573 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:42.552392 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:42.563795 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:42.634173 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:42.653959 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:42.654239 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:42.661819 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:02:42.685587 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:42.689802 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:42.694732 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:42.695829 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:42.698690 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:42.708052 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:42.718159 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:42.738005 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:42.750628 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:42.757000 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:42.775449 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:42.790781 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:02:42.792708 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:42.793346 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:42.808573 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:42.810921 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:42.820626 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:42.822168 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:42.824216 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:42.845085 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:42.907580 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:42.911873 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:02:42.937223 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:42.939034 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:02:42.942134 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:42.950793 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:42.958757 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:42.969689 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:42.990721 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:42.995882 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:43.004820 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:43.005522 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:43.018600 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:02:43.018869 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:43.019450 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:43.021728 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:43.023387 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:43.036861 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:43.152813 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:43.158823 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:43.171955 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:43.185993 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:43.194953 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:43.209961 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:43.214397 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:43.222611 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:43.226817 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:02:43.236449 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:43.241612 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:43.243696 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:43.245882 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:43.248708 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:43.255443 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:43.264393 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:43.268210 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:43.286599 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:43.298530 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:43.299340 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:43.318923 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:43.406131 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:43.407786 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:43.412464 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:02:43.413265 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:43.413664 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:43.426869 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:43.432088 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:43.446831 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:43.500618 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:02:43.520095 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:43.523718 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:43.526363 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:43.528826 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:43.547711 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:43.560027 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:02:43.571204 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:43.579964 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:43.625304 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:02:43.628804 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:43.659458 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:43.662006 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:43.666936 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:43.667692 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:43.689275 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:43.702613 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:43.715332 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:43.720037 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:43.723802 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:43.739061 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:43.740782 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:43.742839 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:43.751588 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:43.817405 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:02:43.818347 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:43.824498 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:43.825122 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:43.825232 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:43.830852 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:43.845536 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:43.895666 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:43.899365 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:43.900771 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:43.905426 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:43.908797 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:43.909488 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:43.953449 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:02:44.025717 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:44.026988 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:44.032673 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:44.033864 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:44.047780 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:44.050683 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:44.055405 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:44.058467 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:44.117491 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:44.122238 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:44.130957 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:02:44.132018 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:44.132019 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:44.136814 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:44.142596 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:44.159780 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:44.227607 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:44.229614 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:44.236598 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:44.240806 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:44.244454 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:44.257439 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:44.270725 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:44.274675 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:44.274856 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:44.279448 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:44.284996 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:44.287148 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:44.305588 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:44.389124 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:44.413697 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:44.418260 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:44.427098 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:44.429885 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:44.434486 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:44.438143 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:44.451112 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:02:44.457854 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:44.458304 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:44.481747 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:44.485423 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:44.487231 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:44.515523 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:44.533015 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:44.570498 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:44.570348 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:44.572057 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:44.578985 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:44.583719 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:44.589633 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:44.591871 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:44.594170 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:44.595157 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:44.627413 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:44.627609 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:44.635296 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:02:44.667105 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:44.667630 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:44.668224 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:44.672897 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:44.684697 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:44.696676 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:44.756093 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:44.758225 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:44.760594 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:44.777884 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:44.778362 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:44.781939 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:44.783488 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:44.840097 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:44.841589 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:44.848865 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:44.871165 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:44.873983 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:44.877460 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:44.936686 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:44.937625 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:44.939337 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:44.941444 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:44.949772 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:44.952183 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:44.996338 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240501 08:02:45.003484 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:45.010890 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:45.015020 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:45.020335 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:45.030161 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:45.041955 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:45.084563 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:45.105255 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:45.107446 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:45.108350 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:45.113076 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:45.117110 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:45.119493 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:45.131682 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:45.143929 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:45.152427 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:45.165973 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:45.170606 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:45.214924 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:45.220961 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:45.223796 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:45.223995 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:45.231842 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:45.247853 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:45.295451 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:45.303602 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:45.303992 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:45.310640 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:45.320256 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:45.321261 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:45.400894 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:45.405107 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:45.421166 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:45.427013 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:45.430794 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:45.431886 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:45.447574 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:45.447892 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:45.458081 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:45.461241 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:45.482697 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:45.483157 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:45.483404 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:45.486295 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:45.493845 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:45.541236 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:45.549707 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:45.554817 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:45.559293 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:45.561033 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:45.568519 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:45.579677 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:45.613651 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:45.632123 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:45.654865 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:45.660270 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:45.661126 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:45.661221 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:45.663462 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:45.664798 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:45.691311 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:45.693871 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:45.727686 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:45.730445 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:45.736603 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:45.737433 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:45.742933 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:45.754966 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:45.770073 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:45.858353 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:45.861371 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:45.861702 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:45.876062 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:45.881400 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:45.895697 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:45.901487 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:45.902654 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:45.911422 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:45.911473 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:45.921728 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:45.938786 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:45.940081 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:46.007748 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:46.026177 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:46.036403 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:46.039811 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:46.046622 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:46.050928 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:46.051565 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:46.053797 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:46.062402 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:02:46.097379 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:02:46.118600 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:46.124867 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:46.126731 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:46.131772 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:46.135308 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:46.138959 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:46.168529 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:46.201400 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:46.206836 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:46.216711 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:46.217444 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:46.231559 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:46.232720 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:46.246052 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:46.317611 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:46.320324 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:46.325263 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:46.325263 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:46.328275 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:46.337446 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:46.340281 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:46.363410 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:46.366164 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:46.369894 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:46.377840 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:46.392910 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:46.420317 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:46.423947 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:46.452876 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:46.476159 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:46.479674 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:46.485973 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:46.487053 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:46.488569 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:46.494566 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:46.500532 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:46.524549 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:46.556768 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:46.557365 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:46.563115 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:46.565420 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:46.568302 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:46.588798 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:46.680397 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:46.681675 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:46.683315 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:46.683921 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:46.697983 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:46.734854 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:02:46.735123 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:46.736181 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:02:46.753165 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:46.753896 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:46.762899 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:46.766285 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:46.772446 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:46.780860 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:46.781513 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:46.859071 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:46.872579 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:46.876996 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:46.885521 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:46.895453 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:46.899653 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:46.900696 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:46.908273 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:46.919500 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:46.925392 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:46.928318 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:46.936374 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:46.940775 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:46.947288 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:46.950901 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:46.971755 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:46.972858 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:46.988389 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:02:46.989883 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 08:02:47.084089 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:47.088855 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:47.093194 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:47.109345 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:47.112815 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:47.118991 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:47.122738 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:47.128388 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:47.131814 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:47.136193 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:47.152941 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:47.159186 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:47.163245 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:47.184329 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:47.184504 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:47.187718 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:47.192545 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:47.201747 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:47.203056 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:47.214733 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:47.253089 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240501 08:02:47.255803 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:47.266731 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:02:47.315203 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:47.331954 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:47.344148 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:47.352527 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:47.358340 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:47.365417 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:47.366739 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:47.368716 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:47.375440 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:47.375545 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:47.377947 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:47.384434 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:47.391367 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:47.403477 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:47.459710 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:47.484189 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:02:47.552651 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:47.554077 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:47.556915 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:47.558737 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 08:02:47.570571 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:47.572696 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:47.579420 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:47.608011 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:47.614291 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:47.621182 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:47.624384 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:47.624769 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:47.636643 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:47.642226 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:47.650234 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:47.653113 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:47.660961 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:47.670135 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:47.671164 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:47.690261 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:47.784171 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:47.798595 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:47.798723 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:47.798851 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:47.799268 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:47.800451 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:47.800441 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:47.820304 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:02:47.906206 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:47.935508 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:47.940841 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:47.942997 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:47.944897 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:47.944906 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:47.947690 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:47.960286 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:47.963585 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:47.964080 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:47.968967 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:47.969925 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:47.976193 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:48.018867 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:48.041800 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:02:48.055804 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:48.068478 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:48.073947 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:48.077065 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:48.080577 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:48.080556 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:48.096194 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:48.173131 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:48.173405 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:48.176940 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:48.186513 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:48.188042 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:48.224113 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:48.229597 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:48.235244 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:48.235244 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:48.241216 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:48.247396 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:48.249907 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:48.270340 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:48.275318 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:48.296350 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:02:48.343770 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:48.371361 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:48.371634 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:48.373754 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:48.379521 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:48.381784 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:48.393800 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:48.411948 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:48.434675 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:02:48.465341 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:48.465905 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:48.469965 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:48.473066 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:48.478662 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:48.492174 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:48.548099 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:48.552397 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:48.553296 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:48.554841 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:48.563128 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:48.566403 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:48.579082 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:48.591670 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:48.599675 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:48.632547 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 08:02:48.639063 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:48.639789 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:48.650883 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:48.652472 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:48.660614 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:48.682667 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:48.725270 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:48.727644 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:48.735126 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:48.742831 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:48.743271 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:48.752681 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:48.786975 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:02:48.816973 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:02:48.863648 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:48.870445 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:48.877835 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:48.895577 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:02:48.900910 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:48.901767 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:48.906097 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:48.922214 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:48.937268 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:48.941040 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:48.960662 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:48.970104 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:48.974174 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:48.986588 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:48.986599 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:48.988477 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:48.999526 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:49.012024 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:49.015064 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:49.015477 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:49.015727 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:49.090480 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:49.107782 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:49.124557 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:49.129309 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:49.129993 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:49.131481 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:49.140946 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:49.160699 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:02:49.172204 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:49.188966 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:49.198546 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:49.198819 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:49.202621 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:49.203923 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:49.207055 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:49.227916 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:49.330215 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:49.334379 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:49.334818 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:49.352702 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:49.360386 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:02:49.385366 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:49.391227 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:49.394913 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:49.395709 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:49.404531 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:49.407838 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:49.410760 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:49.411855 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:49.433295 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:49.438910 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:49.443918 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:49.445286 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:49.445807 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:49.460371 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:49.468454 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:49.488119 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:49.514814 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:49.518908 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:02:49.551188 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:02:49.574182 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:49.575415 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:49.578285 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:49.586648 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:49.589283 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:49.592264 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:49.650480 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:02:49.671877 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:02:49.683239 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:49.684535 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:49.684707 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:49.694653 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:49.696439 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:49.704465 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:49.712857 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:02:49.778627 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:02:49.826982 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:49.833586 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:49.848395 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:49.857178 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:49.872280 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:49.878916 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:49.899807 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:49.901446 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:49.902379 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:49.904369 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:49.909314 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:49.918414 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:02:49.928660 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:49.934675 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:49.948035 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:49.948746 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:49.949492 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:49.955753 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:49.959934 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:49.960471 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:50.051340 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:50.068595 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:50.068886 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:50.069139 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:02:50.072332 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:50.078966 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:50.081352 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:50.085202 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:50.098109 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:50.109221 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:50.165957 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:02:50.194487 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:50.196139 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:50.196192 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:50.196209 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:50.199348 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:50.216041 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:50.328022 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:50.333406 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:50.335995 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:50.350948 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:50.352982 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:50.368147 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:50.368888 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:50.379357 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:50.380156 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:50.380342 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:50.384573 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:50.391093 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:50.391999 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:50.392863 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:50.397442 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:50.412127 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:50.414005 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:50.448690 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:50.463829 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:02:50.469360 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:02:50.474380 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 08:02:50.485102 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:50.504293 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:50.570034 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:50.570371 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:50.572288 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:50.584195 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:50.592908 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:50.594204 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:50.611377 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:50.626751 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:50.631402 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:02:50.634408 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:50.635340 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:50.638010 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:50.640897 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:50.668363 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:02:50.744458 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:50.745088 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:50.754716 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:50.758127 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:50.758677 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:50.781585 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:50.799053 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:50.809767 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:02:50.828985 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:50.833515 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:50.838950 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:50.841114 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:50.844553 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:50.855124 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:50.879142 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:50.904563 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:50.905236 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:50.909752 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:50.929757 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:50.931174 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:50.937604 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:50.938604 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:02:50.959643 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:02:51.008617 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:51.008680 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:51.010349 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:51.016131 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:51.027640 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:51.032662 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:51.087182 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:51.090269 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:51.095065 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:51.097210 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:51.122203 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:51.128405 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:51.133071 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:51.173039 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:51.175551 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:51.184523 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:51.185256 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:51.185725 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:51.187232 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:51.195395 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:51.228156 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:02:51.266181 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:02:51.270610 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:51.292301 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:51.298928 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:51.301190 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:51.318284 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:51.319482 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:51.320000 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:51.324312 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:51.325748 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:51.334897 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:51.351399 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:51.354279 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:51.369470 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:02:51.385901 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:51.407193 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:02:51.431835 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:51.432075 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:51.434530 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:51.434940 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:51.440779 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:51.455116 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:51.549372 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:51.553324 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:51.557034 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:51.563354 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:51.564218 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:51.569741 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:51.585575 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:51.623531 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:02:51.703153 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:51.711997 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:51.716027 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:51.718055 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:51.721283 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:51.727694 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:51.729982 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:51.735455 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:51.736835 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:51.748179 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:51.756340 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:51.759169 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:51.766302 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:51.795418 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:51.805754 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:02:51.862143 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 08:02:51.863905 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:02:51.874018 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:02:51.899052 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:51.905802 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:51.907666 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:51.908268 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:51.922442 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:51.924502 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:51.935617 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:51.935664 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:51.935957 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:51.936794 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:51.969462 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:51.977353 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:52.006475 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:52.042456 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:02:52.044560 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:02:52.060662 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:52.078471 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:52.079259 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:52.081970 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:52.085731 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:52.087602 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:02:52.088793 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:02:52.110594 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:02:52.118247 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:52.118844 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:52.138705 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:52.145159 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:52.147365 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:52.148079 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:52.148860 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:52.189028 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:52.226069 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:52.227267 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:52.232574 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:52.244457 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:52.251693 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:52.258571 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:52.298471 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:52.306021 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240501 08:02:52.320684 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:52.329339 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:52.332036 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:52.336123 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:52.349691 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:52.355605 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:52.360078 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:52.367301 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:52.380326 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:52.385444 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:52.386121 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:52.402658 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:52.411885 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:02:52.450677 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:02:52.463178 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:52.468467 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:52.510908 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:52.513448 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:52.515318 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:52.517203 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:52.518169 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:52.528059 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:52.544974 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:52.593829 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:52.600389 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:52.605763 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:52.618141 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:52.621310 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:52.642410 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:52.654321 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:52.654949 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:52.655040 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:52.675480 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:52.681602 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:02:52.682994 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:02:52.686282 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:52.696694 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:52.786621 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:52.791572 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:52.798676 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:52.799499 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:52.801983 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:52.810698 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:52.832983 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:52.843847 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:52.849251 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:52.852212 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:52.860265 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:52.861428 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:52.862551 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:52.872371 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:52.880142 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:52.883890 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:52.937175 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:52.971236 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:52.971827 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:52.972611 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:52.980966 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:52.981487 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:52.992291 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:52.996510 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:53.044435 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:53.048738 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:53.052714 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:53.059300 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:53.060045 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:53.064141 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:53.107483 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:53.122054 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:53.122315 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:53.133719 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:53.133729 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:53.138084 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:53.153383 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:53.155781 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:53.223134 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:53.225726 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:53.227901 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:53.228221 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:53.233788 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:53.234905 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:53.276072 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:53.289425 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:53.291378 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:53.293834 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:53.295231 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:53.298776 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:53.309561 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:53.311826 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:53.312609 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:53.445763 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:53.450889 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:53.451995 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:53.458649 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:53.463408 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:53.472399 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:53.474265 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:53.550873 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:53.569205 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:53.574308 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:53.577450 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:53.578569 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:53.599977 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:53.609228 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:53.654860 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:53.660641 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:53.667125 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:53.672657 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:53.680155 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:53.683790 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:53.699079 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 08:02:53.773998 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:53.776813 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:53.777348 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:53.783016 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:53.786237 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:53.792668 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:53.802624 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:53.807644 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:02:53.811893 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:53.852859 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:53.866904 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:53.868129 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:53.869105 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:53.897264 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:53.898713 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:53.902307 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:53.904297 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:53.908717 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:53.920428 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:53.927783 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:53.931193 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:53.943260 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:53.953188 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:53.953760 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:53.965425 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:02:54.033609 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:54.053447 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:54.052654 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:54.055233 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:54.059775 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:54.067135 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:54.071621 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:54.077884 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:54.081728 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:54.084432 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:54.092084 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:54.116345 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:54.126580 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:54.241988 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:54.245617 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:54.271596 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:54.272869 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:54.277601 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:54.280344 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:54.281136 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:54.282105 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:02:54.287343 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:54.322230 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:54.332652 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:54.334600 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:54.337255 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:54.359988 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:54.365545 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:54.372982 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:54.372980 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:54.379541 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:54.380903 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:54.384095 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:54.387241 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:54.440124 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:02:54.489771 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:54.508045 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:54.508869 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:54.516351 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:54.516372 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:54.517343 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:54.518198 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:54.531683 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:54.540441 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:54.546051 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:54.550169 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:54.557729 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:54.562687 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:54.572767 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:54.587591 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:54.587783 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:02:54.588511 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:02:54.590691 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:54.702060 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:54.740350 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:54.754994 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:54.758592 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:54.771494 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:54.772619 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:54.786554 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:54.794606 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:54.795986 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:54.802350 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:54.804255 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:54.815049 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:54.822824 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:54.846073 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:54.847959 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:54.850811 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:54.867316 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:54.879253 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:54.891760 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:54.898873 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:54.911809 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:02:54.924923 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:02:54.948202 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:54.978018 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:54.978507 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:54.984237 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:54.984588 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:54.998656 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:55.001027 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:55.003130 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:55.014945 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:55.022753 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:55.031806 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:55.040710 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:55.047029 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:55.051187 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:55.142088 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:55.145316 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:55.145900 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:55.169199 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:55.170645 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:55.179351 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:55.182287 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:55.236413 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:55.236725 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:55.237658 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:55.238631 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:55.239781 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:55.248756 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:55.291384 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:55.306870 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:55.339829 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:55.340685 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:55.341979 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:55.345618 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:55.346011 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:55.350329 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:55.361268 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:55.362861 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:55.396083 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:02:55.419924 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:55.457986 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:55.460500 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:55.462280 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:55.464114 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:55.466511 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:55.468681 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:55.516919 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:55.530311 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:55.534678 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:55.537772 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:55.541455 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:55.557718 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:55.561566 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:55.583717 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:02:55.640266 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:55.644457 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:55.649574 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:55.652443 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:55.653757 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:55.655300 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:55.659668 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:55.706177 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:55.706506 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:55.707353 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:55.722863 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:55.725502 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:55.728319 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:55.742362 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:55.753957 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:55.758064 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:55.808975 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:55.814769 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:55.816187 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:55.816418 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:55.816833 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:55.821285 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:55.901713 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:55.902623 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:55.905786 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:55.917277 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:55.919468 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:55.929200 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:55.931088 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:02:56.004782 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:56.007740 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:56.011263 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:56.012497 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:56.012548 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:56.020602 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:56.057758 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:56.100112 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:56.104243 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:56.109264 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:56.112166 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:56.114006 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:56.128130 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:56.142886 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:56.163704 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:56.167358 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:56.189111 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:56.198047 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:56.202332 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:56.206867 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:56.261202 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:56.261195 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:56.262830 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:56.264822 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:56.273929 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:56.287348 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:56.291626 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:56.310837 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:02:56.312636 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:56.329378 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:56.334585 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:56.334868 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:56.335844 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:56.338065 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:56.340655 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:56.347736 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:02:56.436569 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:56.439954 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:56.443276 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:56.443328 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:56.453600 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:56.463407 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:56.465370 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:02:56.479527 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:56.521533 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:56.521823 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:56.537118 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:56.538739 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:56.553623 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:56.572119 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:56.582875 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:56.600723 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:56.608788 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:56.617463 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:56.631078 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:56.651145 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:56.651995 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:56.679994 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:56.681382 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:56.683733 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:56.686203 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:56.688217 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:56.692214 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:56.741047 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:56.801213 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:56.807320 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:56.808079 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:56.809248 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:56.809410 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:56.814806 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:56.836042 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:02:56.865490 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:02:56.899042 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:56.913134 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:56.920917 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:56.926848 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:56.930524 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:56.939751 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:56.946056 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:02:56.946547 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:56.967075 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:02:56.979290 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:02:56.990662 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:02:56.991497 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:02:57.006630 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:57.017390 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:57.022787 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:57.032261 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:57.036734 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:57.051625 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:57.059036 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:57.062467 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:57.086901 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:57.094996 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:57.105957 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:57.106755 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:57.129101 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:57.130173 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:57.130513 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:57.138471 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:57.139021 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 08:02:57.144353 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:57.150736 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:57.151751 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:02:57.163620 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:57.264772 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:57.266000 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:57.266067 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:57.266944 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:57.271344 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:57.278820 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:57.283655 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:57.317060 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:02:57.322456 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:02:57.323148 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:57.325132 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:02:57.364676 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:02:57.394837 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:57.395232 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:57.395828 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:57.397207 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:57.402618 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:57.408474 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:57.414456 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:57.447114 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:02:57.492460 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:57.496258 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:57.499576 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:57.504741 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:57.509871 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:57.516767 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:57.572314 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:02:57.588001 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:57.589448 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:57.590199 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:57.597177 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:57.599812 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:57.621806 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:57.648583 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:57.652356 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:57.652823 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:57.657856 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:57.658049 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:57.672986 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:57.688376 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:57.700282 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:02:57.706964 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:57.728134 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 08:02:57.748941 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:57.755890 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:57.764349 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:57.770530 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:57.772895 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:57.791195 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:02:57.797781 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:57.839075 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:57.840094 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:57.840710 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:57.841506 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:57.846354 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:57.846587 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:57.945523 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:57.947059 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:57.955528 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:02:57.958006 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:57.959245 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:57.967067 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:57.995501 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:58.026564 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:58.034838 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:02:58.037321 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:58.046041 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:58.055896 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:58.060734 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:58.067698 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:58.068723 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:58.071107 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:58.085139 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:58.088557 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:58.098608 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:58.100593 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:02:58.104084 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:58.112107 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:58.187574 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:58.192324 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:58.193188 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:58.193289 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:58.196204 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:58.215004 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:58.215970 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:02:58.308988 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:58.310827 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:58.311548 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:58.321031 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:58.321020 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:58.323839 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:02:58.330159 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:02:58.341037 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:02:58.345741 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:58.439747 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:58.441179 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:58.450943 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:58.450929 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:58.451130 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:58.461587 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:58.508673 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:58.514384 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:58.516005 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:58.526459 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:58.526479 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:58.528717 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:58.542908 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:58.553586 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:58.556283 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:58.559126 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:58.572539 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:58.576978 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:58.583443 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:58.592700 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:58.665223 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:02:58.703086 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:02:58.711817 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:58.717576 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:58.721467 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:58.725795 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:58.730291 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:58.734766 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:58.739442 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:58.751087 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:58.776407 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:58.780107 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:58.790781 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:58.793843 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240501 08:02:58.817705 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:58.822889 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:02:58.889926 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:58.894477 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:58.895483 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:58.895869 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:58.896580 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:58.906308 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:58.925385 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:02:58.936014 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:02:58.958879 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:58.964339 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:58.978026 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:58.978351 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:58.980036 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:58.988313 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:59.013113 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:59.035562 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:59.035941 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:59.040553 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:59.050307 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:59.053853 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:59.060354 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:59.110648 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:02:59.143951 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:59.146873 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:59.147292 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:59.148087 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:59.148808 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:59.150012 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:02:59.161782 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:59.170617 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:59.260829 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:02:59.271083 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:02:59.272249 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:59.272485 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:59.291122 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:02:59.302129 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:59.303004 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:59.305058 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:59.311837 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:59.319192 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:59.321085 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:59.324421 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:59.330148 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:59.338555 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:59.347056 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:02:59.411036 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:02:59.429301 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:59.432642 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:59.432988 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:59.446002 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:59.457794 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:59.469753 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:59.478348 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:02:59.502792 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:59.503445 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:59.510267 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:59.514672 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:02:59.515645 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:02:59.518630 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:59.539379 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:02:59.540772 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:02:59.603017 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:02:59.619254 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:59.622186 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:59.624320 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:59.633348 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:59.637738 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:59.671316 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:02:59.677042 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:59.690142 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:59.693684 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:59.699631 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:59.700874 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:59.708527 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:59.712394 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:02:59.734746 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:02:59.774498 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:02:59.794235 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:02:59.798574 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:59.799185 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:59.808106 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:59.808403 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:59.813043 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:02:59.883451 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:02:59.896107 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:02:59.896965 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:02:59.901356 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:02:59.904111 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:02:59.911741 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:02:59.911849 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:02:59.925004 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:02:59.958822 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:03:00.010560 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:00.015048 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:00.017767 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:00.018498 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:03:00.018823 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:00.023667 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:00.030670 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:00.042239 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:00.089541 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:00.092281 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:00.098603 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:00.103591 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:00.107431 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:00.114526 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:03:00.130772 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:00.169387 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:03:00.221170 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:00.221180 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:00.234622 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:00.234735 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:00.234776 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:00.236315 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:00.255126 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:03:00.291846 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:00.307651 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:00.318639 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:00.319080 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:00.319208 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:00.319712 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:00.333722 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:00.336357 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:00.336495 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:00.348901 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:00.349267 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:00.369316 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:00.384605 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:00.469761 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:03:00.474853 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:00.482345 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:00.492249 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:00.493211 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:00.493324 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:00.513463 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:00.523825 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:00.536074 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:00.536677 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:00.536700 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:00.546873 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:00.556573 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:00.559499 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:00.601855 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 08:03:00.674799 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:03:00.676106 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:00.680048 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:00.681881 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:00.685590 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:00.693835 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:03:00.694303 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:00.694303 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:00.705740 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:00.709333 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:00.720039 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:00.720825 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:03:00.725742 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:00.739733 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:00.762408 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:00.820483 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:00.830574 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:00.833323 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:00.838599 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:00.840842 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:00.864392 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:00.866562 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:00.884325 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:00.923722 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:03:00.935390 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:00.937860 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:00.939914 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:00.941812 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:00.945530 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:00.959583 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:00.964157 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:00.988991 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:00.993211 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:00.998796 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:00.999545 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:01.013149 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:01.092092 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:01.093283 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:01.102562 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:01.102560 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:01.103994 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:01.110848 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:01.129822 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:01.201331 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:01.201869 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:03:01.203879 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:01.211457 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:01.214119 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:01.215590 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:01.228785 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:01.262269 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:01.274974 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:03:01.292385 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:01.294059 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:01.294327 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:01.296283 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:01.299005 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:01.303186 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:01.319437 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:01.352327 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:01.362710 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:03:01.379652 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:01.379827 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:01.382725 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:01.384815 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:01.390671 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:01.409826 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:01.482344 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:03:01.506565 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:01.509647 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:01.513541 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:01.515352 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:01.517746 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:01.522701 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:01.581007 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:01.596145 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:01.597572 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:01.598827 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:01.616438 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:01.624373 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:01.629551 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:01.684162 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:01.687928 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:01.695348 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:01.698196 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:01.705777 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:01.734642 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:01.764703 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:01.769304 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:03:01.775135 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:01.789841 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:01.791568 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:01.794037 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:01.802210 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:01.805231 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:01.849386 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:03:01.882683 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:01.886533 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:01.890349 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:01.896328 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:01.897581 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:01.911396 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:01.960407 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:01.961321 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:01.964648 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:01.966634 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:01.978811 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:01.984285 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:03:01.988759 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:01.998116 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:02.006906 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:03:02.011467 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:03:02.043638 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:02.046958 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:02.053124 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:02.053318 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:03:02.056313 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:02.057068 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:02.058609 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:02.113499 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:02.127939 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:02.150326 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:02.152985 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:02.158635 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:02.167903 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:02.173662 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:02.176854 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 08:03:02.184247 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:02.186609 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:02.192148 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:02.198963 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:02.201867 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:02.210214 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:02.238849 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:02.251863 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:02.289355 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:03:02.338572 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:02.343078 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:02.348897 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:02.350713 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:02.374801 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:03:02.375053 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:02.383486 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:03:02.386538 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:02.390667 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:03:02.408278 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:02.409827 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:02.415630 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:02.418133 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:03:02.420300 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:02.420953 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:02.426595 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:02.440826 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:02.449514 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:02.460565 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:02.462013 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:03:02.509898 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:02.514616 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:02.519181 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:02.523676 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:02.533624 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:02.537925 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:02.575212 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:03:02.608132 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:02.610479 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:02.614058 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:02.614343 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:02.617079 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:02.624612 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:02.631314 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:02.634938 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:02.643373 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:02.655277 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:02.659318 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:02.673012 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:02.684125 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:02.727599 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:02.763690 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:02.784615 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:02.787369 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:02.794481 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 08:03:02.803258 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:02.806456 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:02.814671 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:02.847615 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:02.848254 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:02.849515 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:02.852656 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:02.859117 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:03:02.861048 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:02.861903 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:02.866158 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:02.930559 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:03:02.955775 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:02.956341 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:02.958366 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:02.961495 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:02.965327 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:02.970727 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:03.029635 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:03.029729 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:03.030921 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:03.034628 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:03.054731 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:03.057448 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:03.188081 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:03.188349 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:03.189886 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:03.192342 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:03.197602 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:03.198040 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:03:03.199251 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:03.247874 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:03.311558 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:03.311707 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:03.312389 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:03.325827 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:03.327371 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:03.344336 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:03.351064 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:03.369381 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:03.387974 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:03.394382 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:03.394441 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:03.394806 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:03.398659 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:03.399536 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:03.464574 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:03:03.469656 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:03.470712 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:03.474287 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:03.477840 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:03.487804 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:03.488245 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:03.499482 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:03.576505 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:03.579334 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:03.582862 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:03.583168 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:03.587482 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:03.590226 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:03.596906 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:03.652647 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:03.659052 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:03.667829 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:03.669759 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:03:03.670506 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:03.675338 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:03.685600 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:03.755332 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:03.770519 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:03:03.774448 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:03.783038 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:03.787479 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:03.787479 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:03.788193 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:03.797585 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:03:03.811076 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:03.811247 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:03.814190 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:03.822886 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:03.845644 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:03.850672 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:03.853435 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:03.880461 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:03.882473 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 08:03:03.892782 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:03.925815 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:03.939260 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:03.941263 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:03.944785 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:03.949337 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:03.955507 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:03.963346 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:04.012655 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:04.018584 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:04.019089 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:04.024013 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:04.024014 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:04.035796 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:04.107851 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:04.108227 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:04.118683 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:04.120286 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:04.120985 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:04.127178 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:04.167695 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:03:04.179335 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:03:04.221302 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:04.226480 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:04.226850 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:04.230182 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:04.230691 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:04.249897 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:04.253749 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:04.255782 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:04.270736 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:04.275538 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:04.285182 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:04.299891 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:04.305316 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:04.312525 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:04.371757 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:04.377056 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:04.378959 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:04.381300 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:04.383550 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:04.397758 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:04.400516 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:04.467021 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:04.471107 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:04.475394 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:04.497851 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:04.497992 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:04.498678 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:04.533454 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:03:04.544922 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:04.554159 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:04.554452 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:04.574957 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:04.582325 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:04.585345 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:04.589329 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:04.590831 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:04.649442 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:04.659518 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:04.662089 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:04.668641 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:04.668942 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:04.670673 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:04.686496 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:04.735882 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:04.770241 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:04.771450 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:04.771656 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:04.773046 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:04.786672 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:04.786952 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:04.797470 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:04.801928 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:04.802610 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:04.804785 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:04.806552 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:03:04.807293 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:04.813066 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:04.815153 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:04.832360 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:04.835629 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:04.893177 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:04.903332 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:04.940678 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:04.945994 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:04.984604 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:04.992841 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:04.996376 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:04.999466 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:05.001085 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:05.009209 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:05.017688 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:05.158650 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:05.159694 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:05.164819 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:05.166095 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:05.166859 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:05.172384 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:05.173213 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:05.213109 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:05.223381 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:05.223448 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:05.228186 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:05.231572 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:05.251458 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:05.253304 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:05.293169 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:05.314558 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:05.365913 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:05.368541 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:05.369493 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:05.385447 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:05.386343 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:05.400715 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:05.402563 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:05.407145 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:05.407639 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:05.425809 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:05.426505 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:05.427029 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:05.427523 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:05.439337 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:05.447147 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:05.456482 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:05.461285 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:05.468326 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:05.469987 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:05.476640 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:05.585660 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:05.613981 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:05.615061 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:05.616303 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:05.622628 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:05.638106 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:05.640718 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:05.640723 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:05.647755 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:05.650373 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:05.660086 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:05.662612 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:05.679342 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:05.699252 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:05.756319 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:03:05.820551 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:05.828320 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:05.838268 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:05.848892 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:05.851912 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:05.852627 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:05.858865 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:05.862335 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:05.877995 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:05.882959 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:05.886587 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:05.896672 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:05.902616 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:05.922966 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:05.931645 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:05.932845 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:05.933959 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:05.944032 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:05.949185 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:05.955464 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:05.979147 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:05.980947 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:06.069612 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:06.071614 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:06.073173 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:06.076251 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:06.087471 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:06.094391 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:06.095978 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:06.109158 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:06.113371 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:06.119354 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:06.120901 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:06.122109 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:06.131956 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:06.215287 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:06.271986 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:06.273530 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:06.275650 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:06.284461 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:06.287166 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:06.299753 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:06.316851 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:06.331591 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:06.341572 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:06.344430 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:06.357568 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:06.360735 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:06.371227 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:06.373683 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:06.374549 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:06.376652 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:06.376683 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:06.389359 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:06.396517 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:06.399048 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:06.407809 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:06.498684 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:03:06.521582 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:06.522737 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:06.534554 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:06.540019 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:06.543460 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:06.548880 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:06.568776 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:06.574023 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:06.574599 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:06.575012 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:06.587556 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:06.588624 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:06.597105 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:06.608589 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:06.616286 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:06.622236 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:06.717211 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:06.721513 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:06.724923 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:06.732264 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:06.735618 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:06.744374 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:06.779590 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:06.790995 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:06.796639 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:06.798703 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:06.822608 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:06.823908 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:06.861069 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:06.872344 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:06.904853 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:06.907186 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:06.909047 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:06.909050 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:06.923564 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:06.935117 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:06.991770 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:03:07.023437 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:07.024339 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:07.024703 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:07.028698 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:07.032552 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:07.065032 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:07.073267 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 08:03:07.073977 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:03:07.093485 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:07.141722 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:07.147063 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:07.156304 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:07.160853 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:07.162146 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:07.186051 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:07.187500 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:07.197188 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:07.198057 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:07.205884 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:07.208451 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:07.213032 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:07.221097 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240501 08:03:07.225163 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:07.241873 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:07.243968 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:07.287436 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:07.289654 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:07.291877 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:07.294813 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:07.297933 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:07.298836 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:07.313230 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:07.377372 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:07.411310 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:07.412958 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:07.413261 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:07.414724 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:07.415319 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:07.420084 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:07.440274 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:03:07.456308 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:07.456532 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:07.475412 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:07.486246 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:07.487840 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:07.493520 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:07.498263 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:07.504297 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:07.505030 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:07.531493 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:07.573103 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:07.596166 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:07.599907 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:07.603137 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:07.604234 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:07.620527 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:07.631953 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:07.666055 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:07.701928 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:07.702091 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:07.702209 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:07.703377 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:07.713660 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:07.728497 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:07.732888 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:07.740782 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:07.747651 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:07.747908 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:07.761991 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:07.762938 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:07.771073 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:07.789332 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:07.806739 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:07.822526 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:03:07.850798 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:07.858084 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:07.858770 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:07.859786 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:07.864815 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:07.867609 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:07.953107 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:07.954190 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:07.963097 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:07.964428 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:07.972662 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:07.974609 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:07.995425 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:07.995915 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:03:08.027109 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:08.039834 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:08.040623 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:08.040712 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:08.041375 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:08.051735 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:08.070158 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:08.078652 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:08.136654 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:08.141849 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:08.146018 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:08.148331 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:08.155251 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:08.172207 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:08.230005 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:08.241007 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:08.243723 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:08.245923 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:08.249707 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:08.254658 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:08.260229 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:08.262003 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:08.270869 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:08.284577 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:08.289052 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:08.294179 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:08.304227 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:08.305087 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:08.386787 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:08.408526 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:08.409242 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:08.410588 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:08.414804 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:08.415062 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:08.415479 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:08.513350 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:08.517443 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:08.520705 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:08.521806 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:08.528368 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:08.528752 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:08.552796 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:08.558795 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:08.591676 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:03:08.601625 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:08.608664 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:08.610576 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:08.610846 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:08.618316 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:08.632835 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:08.635250 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:08.711709 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:08.712168 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:08.712209 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:08.726842 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:08.727994 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:08.745141 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:08.757619 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:08.758846 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:08.775619 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:08.784436 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:08.790549 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:08.791184 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:08.812832 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:08.891990 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:08.892181 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:08.893064 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:08.893325 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:08.896404 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:03:08.899308 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:08.904569 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:08.955361 11598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 08:03:08.977842 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:08.978066 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:08.992823 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:08.993014 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:08.994590 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:09.000960 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:09.008302 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:09.011397 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:09.023939 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:09.039237 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:09.070288 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:09.079950 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:09.080241 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:09.087353 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:09.088452 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:09.088608 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:09.132618 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:09.135416 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:09.165400 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:09.171708 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:09.177286 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:09.186445 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:09.190953 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:09.204090 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:09.237440 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:09.242585 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:09.245678 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:09.246762 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:09.261694 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:09.267488 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:09.276713 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:03:09.331635 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:09.332393 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:09.335300 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:09.337569 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:09.347163 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:09.361975 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:09.378831 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:09.393486 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:09.399258 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:03:09.430838 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:09.434892 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:09.438318 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:09.438376 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:09.438614 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:09.441174 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:09.472857 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:09.474256 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:09.526015 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:09.527734 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:09.533111 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:09.537753 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:09.539086 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:09.539129 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:09.627382 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:09.632011 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:09.633200 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:09.635053 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:09.639276 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:09.647050 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:09.652740 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:09.689188 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:09.699283 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:09.700174 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:09.701032 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:09.705312 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:09.709235 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:09.722537 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:09.731241 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:09.748288 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:03:09.777221 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:09.802132 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:09.804498 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:09.812161 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:09.812546 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:09.813795 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:09.833012 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:03:09.853209 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:09.873025 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:09.897670 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:09.898015 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:09.905961 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:09.906129 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:09.908010 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:09.920099 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:09.952385 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:09.988759 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:09.989391 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:10.019665 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:10.019675 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:10.022794 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:10.025372 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:10.034657 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:10.040652 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:10.085429 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:10.088687 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:10.089601 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:10.091512 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:10.103608 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:10.106710 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:10.127051 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:10.181634 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:03:10.201325 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:10.204753 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:10.207079 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:10.214939 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:10.214975 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:10.217422 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:10.228377 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:03:10.231549 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:10.287051 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:10.290153 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:10.290449 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:10.290693 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:10.297369 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:10.300949 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:10.307124 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:10.383177 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:10.410598 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:10.417165 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:10.432504 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:10.432792 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:10.435403 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:10.453378 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:10.453711 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:10.454807 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:10.458220 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:10.479741 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:10.502599 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:10.503455 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:10.512748 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:10.513783 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:10.518890 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:10.533427 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:10.547535 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:10.554752 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:10.558617 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:10.669418 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:10.671173 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:10.675650 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:10.677078 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:10.680068 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:10.695988 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:10.696540 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:10.699705 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:10.703604 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:10.788964 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:10.805907 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:10.808516 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:10.808565 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:10.811933 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:10.822741 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:03:10.824843 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:10.844169 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:10.937247 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:03:10.940672 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:10.948652 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:10.949115 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:10.961453 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:10.974494 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:10.986363 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:10.991132 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:03:11.006134 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:11.007174 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:11.009938 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:11.015103 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:11.015280 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:11.047822 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:11.070979 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:03:11.076179 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:11.077922 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:11.080215 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:11.084846 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:11.086223 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:11.096427 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:11.112360 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:11.120038 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:03:11.177917 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:11.179805 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:11.181075 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:11.186161 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:11.190675 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:11.196609 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:11.198446 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:11.255566 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:11.256865 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:11.261668 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:11.270007 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:11.271948 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:11.273737 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:11.324138 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:11.337267 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:03:11.352073 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:03:11.360752 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:11.369591 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:11.371825 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:11.374632 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:11.378930 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:11.408092 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:11.470542 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:11.474507 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:11.475761 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:11.479185 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:11.480559 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:03:11.485519 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:11.489233 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:11.492094 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:11.494910 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:03:11.500962 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:11.502243 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:11.506134 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:11.515061 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:11.521296 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:11.535316 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:11.621503 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:11.621790 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:11.621968 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:11.626596 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:11.626897 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:11.635728 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:11.739120 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:03:11.758355 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:11.763743 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:11.768940 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:11.771468 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:11.779064 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:11.782438 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:11.797216 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:11.800246 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:11.850556 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:11.881623 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:03:11.898981 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:11.904687 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:11.911998 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:11.934057 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:11.956404 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:11.959888 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:11.968840 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:11.969774 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:11.972457 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:11.975584 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:11.978482 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:11.979985 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:11.991677 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:11.993094 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:11.993309 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:11.996255 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:12.011027 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:12.023025 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:12.027594 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:12.031189 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:12.061298 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:12.112215 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:03:12.145936 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:12.149471 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:03:12.149569 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:12.150854 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:03:12.150949 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:12.160467 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:12.163157 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:12.164628 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:12.201040 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:12.208854 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:12.217093 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:12.221534 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:12.232548 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:12.233861 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:12.252166 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:12.273190 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:12.329177 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:12.342775 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:12.347400 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:12.357702 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:12.363128 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:12.368428 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240501 08:03:12.368518 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:12.372295 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:03:12.378719 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:12.383056 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:12.418728 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:03:12.443621 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:12.444603 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:12.445156 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:12.455274 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 08:03:12.456672 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:12.461416 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:12.470618 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:12.471428 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:12.483963 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:12.486357 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:12.495723 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:12.497509 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:12.510739 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:12.531978 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:03:12.611780 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:12.615172 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:12.618623 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:12.620946 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:12.627872 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:12.636720 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:12.679534 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:12.686601 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:03:12.699255 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:12.709177 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:12.732975 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:03:12.761775 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:12.762461 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:12.774086 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:12.787606 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:12.797678 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:12.807049 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:12.836706 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:12.891036 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:12.892504 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:12.907568 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:12.913949 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:12.914147 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:12.925227 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:12.951460 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:12.955917 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:12.958782 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:12.960343 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:12.974283 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:12.974587 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:03:12.985673 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:13.044139 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:13.050956 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:13.051782 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:13.052137 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:13.055895 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:13.060865 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:13.080685 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240501 08:03:13.102878 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:03:13.111570 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:13.118213 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:13.118321 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:13.122113 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:13.123867 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:13.130867 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:13.150998 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:13.253741 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:13.267380 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:13.280088 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:13.281200 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:13.283520 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:03:13.283593 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:13.290625 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:13.301038 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:13.306262 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:13.317839 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:13.330559 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:13.331393 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:13.362473 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:13.379860 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:13.381318 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:13.384008 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:13.385088 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:13.386881 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:13.398093 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:13.402678 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:13.430795 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:13.498991 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:13.502957 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:13.507112 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:13.507445 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:13.507485 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:13.516781 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:13.564700 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:13.615172 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:13.615329 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:13.624162 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:13.627828 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:03:13.628873 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:13.629036 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:13.647783 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:13.655355 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:13.678695 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:13.720316 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:13.744567 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:13.746694 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:13.748242 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:13.750945 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:13.760623 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:13.771935 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:13.772255 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:13.782235 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:13.831002 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:03:13.839056 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:13.843840 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:13.848747 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:13.857355 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:13.860354 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:13.863026 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:13.892371 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:13.893039 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:13.900871 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:13.919170 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:13.919517 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:13.920260 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:13.981218 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:03:13.981887 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:14.021194 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:14.027364 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:14.027820 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:14.031961 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:14.043350 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:14.062546 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:14.069161 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:14.070304 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:03:14.071425 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:14.099421 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:14.101958 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:14.107620 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:14.108580 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:14.118613 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:14.210116 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:14.214968 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:03:14.220223 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:14.228094 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:14.236048 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:14.248502 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:14.249969 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:14.260354 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:14.268625 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:14.277290 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:14.284826 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:14.290541 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:14.293042 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:14.308676 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:14.322625 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:14.356485 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:14.359809 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:14.359864 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:14.360159 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:14.363832 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:14.366616 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:14.451532 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:14.452365 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:14.452771 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:14.457623 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:14.459937 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:14.460158 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:14.460716 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:14.468067 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:03:14.469503 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:14.572721 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:14.574436 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:14.581808 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:14.583395 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:03:14.586870 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:14.587113 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:14.595700 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:14.613497 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:14.617439 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:14.648361 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:14.660040 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:14.677217 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:14.679090 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:14.681674 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:14.685783 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:14.706634 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:14.734278 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:14.755201 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:14.757753 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:14.777594 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:14.779891 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:14.782578 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:14.813334 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:14.838060 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:14.843709 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:14.846145 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:14.848273 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:14.849875 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:14.872331 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:14.878612 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:14.890707 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:14.952476 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:14.984462 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:14.986546 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:14.987102 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:14.988667 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:14.989540 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:14.991026 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:15.001809 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:15.036269 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:15.042124 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:15.042670 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:15.043238 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:15.079813 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:15.084733 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:15.087230 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:15.122287 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:15.128162 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:15.132644 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:15.132897 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:15.139771 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:15.170465 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:15.187314 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:15.208061 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:03:15.209124 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:15.212884 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:15.214905 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:15.218097 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:15.225339 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:15.228577 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:15.259851 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:15.279422 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:15.324331 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:15.325605 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:15.330125 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:15.333062 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:15.333158 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:15.335459 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:15.347524 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:15.413717 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:15.437520 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:15.476250 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:15.479288 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:15.480227 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:15.482053 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:15.494127 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:15.514816 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:15.547734 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:15.549862 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:15.555296 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:15.558570 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:15.560895 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:15.565831 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:15.565979 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:15.574122 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:15.582604 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:15.585948 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:15.613284 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:15.618800 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:15.619776 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:15.626825 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:15.707878 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:15.709471 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:15.717898 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:15.721971 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:15.723093 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:15.723794 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:15.740854 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:15.756564 11467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.133:42805, user_credentials={real_user=slave}} blocked reactor thread for 44873.7us
I20240501 08:03:15.804481 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:15.805384 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:15.807027 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:15.807750 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:15.813822 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:15.820855 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:15.830873 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:15.872648 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:15.894598 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:03:15.931941 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:15.942040 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:15.944114 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:15.949720 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:15.961108 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 08:03:15.962615 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:03:15.972743 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:15.973599 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:03:15.974448 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:16.003710 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:16.003672 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:16.017513 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:16.022217 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:16.023775 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:16.025350 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:16.039680 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:16.061936 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:16.070096 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:16.078178 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:16.078455 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:16.082610 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:16.174470 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:16.202142 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:16.207796 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:16.208806 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:16.208897 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:16.209519 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:16.211206 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:16.216089 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:16.225373 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:16.229912 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:16.230196 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:16.243996 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:16.247686 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:16.255399 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:16.277530 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:16.314831 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:16.354951 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:03:16.399099 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:16.406107 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:16.409204 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:16.411139 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:16.414366 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:16.414829 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:16.416776 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:16.418303 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:16.423486 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:16.428220 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:16.443246 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:16.444814 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:16.450107 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:16.469966 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:16.489646 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:16.581624 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:16.583781 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:16.586227 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:16.588057 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:16.606065 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:16.607795 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:16.614722 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:16.622339 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:16.629770 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:16.629753 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:16.630570 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:16.640882 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:16.673367 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:16.711998 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:16.714174 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:16.808225 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:16.812721 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:16.812933 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:16.814122 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:16.821509 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:16.832921 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:16.844209 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:16.848042 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:16.850663 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:16.853155 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:16.855273 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:16.868937 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:16.870862 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:16.877219 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:16.883797 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:16.887899 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:16.890049 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:16.901499 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:16.910641 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:16.915295 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:16.931363 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:16.988757 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:17.031982 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:17.040747 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:17.042032 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:17.045919 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:17.049999 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:17.059072 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:17.060042 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:17.153234 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:17.153625 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:17.155021 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:17.154932 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:17.167377 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:17.170346 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:17.210595 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:17.234825 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:17.243898 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:17.267899 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:03:17.269531 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 08:03:17.269863 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:17.274534 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:17.274820 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:17.288030 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:17.296114 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:17.308629 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:17.327396 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:17.331423 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:17.337818 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:03:17.342866 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:17.346634 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:17.354645 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:17.369588 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:17.372373 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 08:03:17.400089 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:17.438529 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:17.438906 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:17.439810 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:17.443579 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:17.443945 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:17.459928 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:03:17.461133 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:17.461565 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:17.472275 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:17.490195 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:03:17.511749 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:17.518559 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:17.520464 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:17.528122 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:17.528664 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:17.528913 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:17.621634 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:17.626215 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:17.628854 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:17.629513 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:17.635608 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:17.640029 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:17.640509 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:17.714264 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:17.739948 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:17.740466 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:17.748099 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:17.755667 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:17.760761 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:17.772826 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:17.773275 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:17.783345 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:17.793341 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:17.803718 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:17.812057 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:17.830158 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:17.833178 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:17.862102 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:17.906507 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:17.912901 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:17.912987 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:17.918634 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:17.920724 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:17.953217 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:17.961973 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:03:17.966928 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:17.978580 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:17.981019 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:17.981148 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:17.988811 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:17.997584 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:18.003787 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:18.028787 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:03:18.085232 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:18.087677 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 83 similar messages]
I20240501 08:03:18.088742 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:18.106248 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:18.107512 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:18.120357 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:18.124100 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:18.169281 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:18.174299 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:18.176961 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:18.177404 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:18.192023 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:18.201373 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:18.231063 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 08:03:18.262199 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:18.265940 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:18.266991 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:18.270707 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:18.275915 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:18.287729 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:18.318022 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:18.352366 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:18.359351 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:18.360989 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:18.369380 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:18.370534 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:18.381398 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:18.407217 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:03:18.475697 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:18.478091 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:18.478667 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:18.487619 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:03:18.488021 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:18.497151 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:18.502373 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:18.503198 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:18.536458 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:18.536724 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:18.555538 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:18.559438 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:18.569612 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:18.573410 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:18.575241 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:18.581161 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:18.658895 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:18.662803 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:18.664376 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:18.669845 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:18.679594 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:18.688874 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:18.723474 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:18.740002 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:18.742877 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:18.746542 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:18.747225 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:18.761021 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:18.771965 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:03:18.772984 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:18.781109 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:18.894522 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:18.897377 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:18.905018 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:18.908041 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:18.920567 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:18.930318 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:18.942826 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:18.944479 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:03:18.946389 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:18.946503 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:18.946952 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:18.953518 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:18.977918 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:18.984877 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:19.000808 11598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:03:19.020989 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:19.029417 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:19.084822 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:19.091351 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:03:19.097216 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:19.100903 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:19.108055 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:19.110200 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:19.114483 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:19.127626 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:19.128692 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:19.139288 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:19.144690 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:19.144973 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:19.147994 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:19.153339 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:19.159083 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:19.163398 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:19.184444 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:19.196146 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:19.202957 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:19.204820 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:19.227209 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:19.230762 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:19.310964 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:03:19.356999 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:19.369112 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:19.370142 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:19.371940 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:19.373618 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:19.375650 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:19.383234 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:19.387060 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:19.395602 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:19.398979 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:19.399703 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:19.406850 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:19.413185 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:19.413879 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:19.467991 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:19.545497 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:19.548328 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:19.548514 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:19.551467 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:19.558494 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:19.558768 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:19.562081 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:03:19.570639 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:19.570758 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:19.573549 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:19.576120 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:19.578598 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:19.582525 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:19.600137 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:19.614460 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:19.674463 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:19.681233 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:19.681232 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:19.681536 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:19.684363 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:19.702855 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:19.733994 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:19.774231 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:19.786036 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:19.790568 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:19.799818 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:19.800071 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:19.805253 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:19.809100 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:19.877635 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:19.886752 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:19.886874 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:19.888566 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:19.894133 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:19.910122 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:19.928731 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 08:03:19.983769 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:19.987890 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:19.991485 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:19.992491 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:19.997630 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:20.007701 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:20.011516 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:03:20.026880 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:20.032140 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:20.033638 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:20.040915 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:20.041247 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:20.041848 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:20.057780 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:20.104820 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:03:20.126221 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:20.127528 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:20.133541 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:20.139825 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:20.140517 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:20.151777 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:20.163837 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:20.204643 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:20.219897 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:20.220268 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:20.221621 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:20.223520 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:20.233184 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:20.238226 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:20.311676 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:20.353307 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:20.382829 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:20.385356 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:20.387322 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:20.387974 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:20.398470 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:20.399672 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:20.403350 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:20.422358 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:20.479590 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:20.481328 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:20.482199 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:20.485149 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:20.490602 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:20.498513 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:20.593587 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:20.594815 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:20.595093 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:20.596975 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:20.602576 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:20.605464 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:20.607492 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:20.641618 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:20.670302 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:20.670576 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:20.671438 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:20.672379 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:20.681980 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:20.682307 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:20.712675 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 08:03:20.766841 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:03:20.807693 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:20.812940 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:20.814522 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:20.815409 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:20.830400 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:20.855222 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:20.857326 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:20.868036 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:20.872908 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:20.892779 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:20.902623 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:20.903136 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:20.907303 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 17 similar messages]
I20240501 08:03:20.910629 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:20.911803 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:20.919025 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:20.926653 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:20.928900 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:20.935478 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:20.935657 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:20.937520 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:21.000099 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:21.054776 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:21.055631 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:21.056339 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:21.062016 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:21.072271 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:21.082633 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:21.093505 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:21.093683 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:21.097168 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:21.111716 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:21.124825 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:21.126775 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:21.130692 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:21.212847 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:21.234278 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:21.235713 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:21.245682 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:21.250332 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:21.277627 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:21.283138 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:21.292204 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:21.296044 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:21.300403 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:21.308848 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:21.310025 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:21.313889 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:21.316604 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:21.334331 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:21.343909 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:21.348318 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:21.352250 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:21.359333 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:21.384126 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:21.392165 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:21.397104 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:21.406069 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:21.406184 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:21.497002 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:21.499569 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:21.502527 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:21.505882 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:21.518564 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:21.540649 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:21.541304 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:21.546207 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:21.547752 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:21.561729 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:21.571031 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:21.574685 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:21.583623 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:21.642500 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:21.673413 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:21.676054 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:21.683728 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:21.688314 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:21.716686 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:21.733386 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:21.733690 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:21.758703 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:21.769352 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:21.774211 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:21.787403 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:21.789958 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:21.791615 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:21.799180 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:21.800750 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:21.804111 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:21.815547 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:21.818338 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:21.819132 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:21.823581 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:21.833889 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:21.836840 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:21.839413 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:21.840488 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:21.846602 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:21.914523 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:21.935133 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:21.935820 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:21.939746 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:21.947427 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:21.959518 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:21.965198 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:21.997210 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:22.003077 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:22.005126 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:22.011665 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:22.017510 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:22.026480 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:22.041565 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:22.066355 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:22.163900 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:22.165197 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:22.172703 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:22.183228 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:22.184011 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:22.193610 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:22.194106 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:22.286491 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:22.287539 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:22.289875 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:22.300491 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:22.305161 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:22.307025 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:22.315199 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:22.335100 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:22.358916 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:03:22.372618 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:22.375942 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:22.380059 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:22.385442 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:22.394886 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:22.397500 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:22.397770 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:22.400660 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:03:22.459616 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:22.464627 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:22.465286 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 08:03:22.466883 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:22.477787 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:22.495643 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:22.503114 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:22.507948 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 08:03:22.552647 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:22.558384 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:22.560331 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:22.576911 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:22.577664 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:22.616478 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:22.642741 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:22.655115 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:22.656603 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:22.669714 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:22.669764 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:22.676991 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:22.692198 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:22.694895 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:03:22.697835 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:22.718709 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:22.718945 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:22.730165 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:22.740734 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:22.743395 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:22.747728 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:22.804203 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:22.822358 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:22.848752 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:22.853070 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:22.855834 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:22.876487 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:22.883405 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:22.894912 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:22.895001 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:22.903791 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:22.910576 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:22.915303 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:22.915966 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:22.924062 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:22.925189 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:22.926595 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:22.960191 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:22.961469 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:23.064275 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:23.064975 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:23.074575 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:23.075762 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:23.076236 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:23.084426 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:23.086867 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:23.106889 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:23.116907 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:23.117686 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240501 08:03:23.119844 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:23.122313 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:23.145833 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:23.147045 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:23.147059 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:23.163558 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:23.167354 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:23.184840 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:23.186971 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:23.253451 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:23.281159 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:23.292343 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:23.297806 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:23.303712 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:23.304164 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:23.307718 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:23.310457 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:23.335719 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:23.347415 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:23.351408 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:23.352619 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:23.372568 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:03:23.382503 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:23.386090 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:23.425837 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:23.476342 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:23.476650 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:23.478255 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:23.492548 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:23.498615 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:23.518348 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:23.523787 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:23.540452 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:23.544620 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:23.547622 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:23.548910 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:23.552600 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 08:03:23.561383 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:23.581545 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:23.581813 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:23.584702 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:23.589257 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:23.597211 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:23.605238 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:23.606503 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:23.614210 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:23.635690 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:23.655647 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:23.673871 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:23.741330 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:23.741515 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:23.743299 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:23.748204 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:23.765367 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:23.766511 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:23.876334 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:23.883131 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:03:23.888056 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:23.888373 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:23.888677 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:23.889001 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:23.911316 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:23.915988 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:23.930322 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:23.936131 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:23.940681 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:03:23.941097 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:23.944272 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:23.946184 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:23.954403 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:23.991150 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:24.012583 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:03:24.032682 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:24.036765 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:24.038756 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:24.044325 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:24.045291 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:24.046777 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:24.116489 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:24.117143 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:24.117208 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:24.119688 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:24.129173 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:24.130282 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:24.136433 11598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:03:24.193814 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:24.222151 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:24.223232 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:24.226652 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:24.245708 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:24.250769 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:24.250886 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:24.251662 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:24.254439 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:24.271243 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:24.310112 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:24.310767 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:24.317270 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:24.317294 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:24.319793 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:03:24.320758 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:24.325112 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:24.345175 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:24.349779 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:24.351357 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:24.415592 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:24.416747 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:24.430272 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:24.435675 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:24.453245 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:24.461796 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:24.470727 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:24.488947 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:24.490844 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:24.491490 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:24.493934 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:24.502630 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:24.507369 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:24.526232 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:24.570297 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:24.584829 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:24.585556 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:24.594482 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:24.594801 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:24.595614 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:24.596357 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:24.633930 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:24.657816 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:24.676337 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:24.677426 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:24.678674 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:24.683575 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:24.686964 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:24.698966 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:03:24.711398 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:24.746498 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:24.788841 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:24.789011 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:24.790812 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:24.827708 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:24.827791 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:24.828595 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:24.888473 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:24.902302 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:24.905668 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:24.905948 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:24.907878 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:24.912986 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:24.948359 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:24.950444 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:24.961246 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:24.962018 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:24.964929 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:24.978998 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:25.035948 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:25.037030 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:25.040748 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:25.042507 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:25.043326 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:25.048216 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:25.074733 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:25.122321 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:25.123812 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:25.127027 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:25.134689 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:25.146693 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:25.147670 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:25.148171 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 08:03:25.221233 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:25.224788 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:25.226127 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:25.226389 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:25.226469 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:25.235975 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:25.242671 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:25.307027 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:25.312184 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:25.317266 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:25.323035 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:25.323238 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:25.324620 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:25.335952 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:25.343796 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:03:25.362900 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:25.405874 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:25.408015 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:25.409010 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:25.410511 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:25.421968 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:25.423910 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:25.451781 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:25.499286 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:25.504629 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:25.508249 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:25.508256 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:25.509335 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:25.510273 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:25.519286 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:25.520357 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:25.533486 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:03:25.586239 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:25.591364 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:25.599462 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:25.599823 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:25.616358 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:25.624672 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:25.672451 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:25.677237 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:25.679596 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:25.681665 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:25.684887 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:25.701992 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:25.789760 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:25.795763 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:25.795887 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:25.797425 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:25.804638 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:25.806149 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:25.845182 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:25.852514 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:25.876250 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:25.876969 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:25.877130 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:25.881556 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:03:25.899969 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:25.913676 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:25.928890 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:25.930692 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:25.933135 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:25.934329 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:25.934537 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:03:25.948597 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:25.955667 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:25.959197 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:25.973188 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:26.015982 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 08:03:26.031805 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:03:26.058699 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:26.058996 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:26.061230 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:26.064155 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:26.072475 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:26.072814 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:26.089612 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:26.107293 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:03:26.176482 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:26.181988 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:26.194602 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:26.195906 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:26.207758 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:26.210786 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:26.238562 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:26.238658 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:26.240499 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:26.241905 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:03:26.264839 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:26.265583 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:26.267356 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:26.267438 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:26.311368 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:26.330827 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:03:26.349512 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:26.350137 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:26.352417 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:26.353428 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:26.367069 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:26.385355 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:26.420053 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:26.424093 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:26.429524 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:26.434176 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:26.446585 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:26.452288 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:26.522908 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:26.533725 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:26.535163 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:26.538406 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:26.550376 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:26.550817 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:26.569432 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:26.599048 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:26.612927 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:26.613441 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:26.615239 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:26.617841 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:26.634212 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:26.640916 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:26.641638 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:26.661657 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:26.675979 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:03:26.691495 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:26.694990 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:26.694990 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:26.700021 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:26.706580 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:26.710656 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:26.711352 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:26.725123 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:26.755859 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:26.761193 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:03:26.786885 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:26.792500 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:26.792511 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:26.792609 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:26.793406 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:26.796669 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:26.886665 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:26.900808 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:26.906978 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:26.909415 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:26.909709 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:26.910344 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:03:26.910377 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:26.925029 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:26.992525 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:26.996089 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:27.004132 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:27.006510 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:27.010018 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:27.032179 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:27.082612 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:03:27.088771 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:27.097105 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:27.097162 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:27.100368 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:27.100651 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:27.123034 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:27.129285 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:03:27.134687 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:27.135581 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:27.144958 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:27.145802 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:27.146390 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:27.156512 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:27.174896 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:27.187016 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:27.232668 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:03:27.263439 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:27.265113 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:27.267908 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:27.268436 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:27.272629 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:27.279446 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:27.339036 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:27.339964 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:27.346668 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:27.351167 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:27.352525 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:27.378258 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:27.381263 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:27.461753 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:27.462949 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:27.472124 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:27.475442 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:27.479902 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:27.491645 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:27.492388 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:27.509212 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:27.514936 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:27.516237 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:27.531323 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:27.536147 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 08:03:27.550812 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:27.609262 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:27.612538 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:27.615010 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:27.617065 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:27.619705 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:27.638103 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:27.650597 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:03:27.703044 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 08:03:27.708359 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:27.709808 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:27.719368 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:27.722349 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:27.734813 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:27.746613 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:27.768085 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:03:27.768776 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:27.771783 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:27.781406 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:27.806017 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:27.811738 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:27.816414 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:27.821830 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:27.822579 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:27.825035 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:27.831588 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:27.833073 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:03:27.908661 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:27.914142 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:27.919196 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:27.920953 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:27.923032 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:27.949978 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:27.976455 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 08:03:27.990299 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:03:27.991058 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:27.992600 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:27.993659 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:28.003253 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:28.010466 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:28.012506 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:28.111502 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:28.113667 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:28.115916 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:28.116604 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:28.124022 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:28.127353 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:28.156584 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:03:28.172456 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:03:28.195202 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:28.197752 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:28.200030 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:28.200899 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:28.203318 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:28.222302 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:28.227797 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:28.243309 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:28.253863 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:03:28.303253 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:28.306936 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:28.311954 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:28.312796 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:28.328243 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:28.335316 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:28.373982 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:28.395262 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:28.395602 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:28.398258 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:28.399572 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:28.411625 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:28.412150 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:28.414999 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:28.424840 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:03:28.443014 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:28.443291 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:03:28.448715 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:28.449676 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:28.455500 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:28.474673 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:28.574357 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:28.579826 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:28.580967 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:28.581838 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:28.583446 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:28.585971 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:28.608651 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:03:28.686894 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:28.696143 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:28.711267 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:28.711717 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:28.713385 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:28.715946 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:28.735008 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:28.764710 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:28.787119 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:28.793442 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:28.795127 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:28.803180 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:28.809356 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:28.824317 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:28.839497 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:28.842767 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:28.889235 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:28.913902 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:28.915786 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:28.915947 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:28.917521 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:28.923507 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:28.934592 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:28.953639 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:28.957901 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:28.959064 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:28.969288 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:28.971698 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:03:28.977442 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:28.979272 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:03:28.985062 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:29.046367 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:03:29.059322 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:29.062882 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:29.066979 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:29.068610 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:29.070119 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:29.079627 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:29.085261 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:29.138186 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:29.138686 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:29.139423 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:29.140040 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:29.150159 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:29.155627 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:29.180166 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:29.240731 11598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:03:29.253652 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:29.256459 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:29.257050 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:29.257776 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:29.263427 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:29.290691 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:29.296525 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:29.331707 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:29.336704 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:29.337169 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:29.346468 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:29.346946 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:29.349929 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:29.370771 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:29.375705 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:29.392010 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:29.411443 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:29.432476 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:29.433295 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:29.433508 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:29.440280 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:29.444680 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:29.449172 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:29.462790 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:03:29.514468 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:29.519454 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:29.520332 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:29.520792 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:29.528723 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:29.545750 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:29.552459 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:29.589676 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:29.593315 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:29.594067 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:29.597414 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:29.602350 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:29.627991 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:29.668407 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:29.691144 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:29.698985 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:29.701062 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:29.713773 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:29.724507 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:03:29.738235 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:29.740756 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:29.742762 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:29.786990 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:29.793442 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:29.799793 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:29.805672 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:29.808490 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:03:29.809110 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:29.815511 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:03:29.822609 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:29.859483 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:29.861979 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:29.865494 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:29.869493 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:29.874147 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:29.877429 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:29.918594 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:29.975821 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:29.977228 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:29.979274 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:29.979365 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:29.981676 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:29.990962 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:30.032178 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:30.081085 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:30.086795 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:30.087059 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:30.087337 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:30.089705 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:30.093523 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:30.138278 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 08:03:30.152540 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 08:03:30.217782 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:30.219686 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:30.223670 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:30.226847 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:30.228888 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:30.231467 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:30.234568 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:30.236266 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:30.244302 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:30.245342 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:30.248096 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:30.251816 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:30.260512 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:30.262960 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:30.364717 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:30.393203 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:30.393766 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:30.407178 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:30.409771 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:30.418025 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:30.418607 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:30.450006 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:30.450209 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:30.452920 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:30.454970 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:30.485265 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:30.490125 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:30.509407 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:03:30.577868 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:30.589874 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:30.589977 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:30.592290 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:30.596621 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:30.600800 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:30.621714 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:03:30.624285 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:30.627163 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:30.633368 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:30.643306 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:30.647133 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:30.652988 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:30.670560 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:30.687745 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:03:30.744728 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:30.745904 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:30.747028 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:30.747552 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:30.747557 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:30.755993 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:30.779655 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:30.788403 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:30.829640 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:30.832713 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:30.846094 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:30.856142 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:30.860397 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:30.861740 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:30.887498 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:30.894191 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:30.895973 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:30.905148 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:30.909878 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:30.910845 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:30.911095 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:30.918788 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:30.948017 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:31.020138 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:31.027079 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:31.039255 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:03:31.043987 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:31.047065 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:31.050812 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:31.051803 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:31.054773 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:31.074816 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:31.077448 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:31.091586 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:31.094024 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:31.101420 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:31.112155 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:03:31.142284 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:31.215337 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:31.234921 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:31.238740 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:31.247220 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:31.247903 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:31.258105 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:31.259598 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:31.262184 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:31.267597 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:31.269951 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:31.273128 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:31.311480 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:31.329699 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:31.337953 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:31.351255 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:31.363384 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:31.363945 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:31.364537 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:31.368580 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:31.376587 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:31.445154 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:31.455229 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:31.455940 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:31.460064 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:31.461059 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:31.462158 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:31.462502 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:31.492477 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:31.519371 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:03:31.530930 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:31.530942 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:31.535203 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:31.541975 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:31.547564 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:31.549539 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:31.579857 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:03:31.629505 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:31.630097 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:31.637454 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:31.638267 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:03:31.644557 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:31.649663 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:31.658131 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:31.703670 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:31.706456 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:31.706532 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:31.721683 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:31.723021 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:31.723820 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:31.725675 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:31.735529 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:31.759907 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:31.799603 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:31.804463 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:31.862919 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:31.863022 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:31.864053 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:31.864943 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:31.870203 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:31.874611 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:31.899884 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:31.918388 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:31.954475 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:31.980886 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:31.986977 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:31.988716 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:31.991312 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:31.999246 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:32.009274 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:32.030184 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:32.030687 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:32.033169 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:32.039011 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:32.039290 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:32.045800 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:32.088037 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:32.095624 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:03:32.153844 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:32.154163 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:32.158887 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:32.163525 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:32.184144 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:32.191504 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:32.194877 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:32.248430 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:32.250741 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:32.255736 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:32.265329 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:32.266219 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:32.283114 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:32.297053 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:32.300752 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:32.301651 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:32.306041 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:32.306649 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:32.312580 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:32.421692 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:32.432948 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:32.439800 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:32.446735 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:32.452203 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:32.455562 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:32.470331 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:32.471836 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:32.486991 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:32.506729 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:32.511019 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:32.522219 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:32.535028 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:32.539870 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240501 08:03:32.543648 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:32.545136 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:32.580883 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:03:32.640882 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:32.641440 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:32.642100 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:32.646070 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:32.647688 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:32.650346 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:32.655308 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:32.719391 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:32.731933 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:32.732609 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:32.734570 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:32.738797 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:32.743585 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:03:32.747849 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:32.761795 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:32.763298 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:32.766178 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:32.767097 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:32.767359 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:32.776441 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:32.782740 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 08:03:32.815006 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:32.918407 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:03:32.919950 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 08:03:32.920533 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:32.922883 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:32.925326 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:32.944451 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:32.947705 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:32.958359 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:32.961969 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:32.984124 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:03:33.000015 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:33.000451 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:33.005860 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:33.013466 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:33.013515 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:33.015213 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:33.029804 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:33.081234 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:33.094970 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:33.095675 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:33.103307 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:33.110009 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:33.128149 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:33.138062 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:33.147327 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:33.151087 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:33.157722 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:33.167410 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:33.181319 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:33.183972 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240501 08:03:33.186509 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:33.202930 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:33.250980 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:33.263064 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:33.264885 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:33.274108 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:33.277542 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:33.292469 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:33.303874 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:33.336133 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:33.338953 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:33.340255 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:33.344583 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:33.349324 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:33.363904 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:03:33.372921 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:33.416080 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 08:03:33.428280 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:33.438274 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:33.438817 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:33.442023 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:33.443370 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:33.448308 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:33.455318 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:03:33.458779 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:33.519955 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:33.524492 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:33.526072 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:33.530614 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:33.530642 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:33.531584 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:33.584817 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:33.621182 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:33.688555 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:33.689747 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:33.689872 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:33.689951 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:33.692166 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:33.715688 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:33.718804 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:33.827436 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:03:33.838663 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:33.838992 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:33.839974 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:33.846215 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:33.847260 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:33.868144 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:33.872957 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:33.878621 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:33.887204 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:33.887495 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:33.890683 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:33.897205 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:33.913069 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:33.948972 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:33.967564 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:33.993791 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:33.996857 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:34.005221 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:34.005446 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:34.009544 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:34.017863 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:34.043556 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:34.047783 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:34.086835 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:34.089478 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:34.091604 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:34.095702 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:34.097582 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:34.115108 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:34.121995 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:34.129710 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:34.136279 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:34.142330 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:34.146983 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:34.149441 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:03:34.171387 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:34.191311 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:34.191548 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:34.208947 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:34.211925 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:03:34.272744 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:34.279204 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:34.286501 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:34.286784 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:34.313704 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:34.321352 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:34.335256 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 08:03:34.366580 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:34.368075 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:34.368450 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:34.372665 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:34.378407 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:34.380534 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:34.398376 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:34.440305 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:34.455379 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:34.457288 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:34.460654 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:34.463589 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:34.474889 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:34.479205 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:34.506230 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:34.531934 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:34.538380 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:34.539078 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:34.540571 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:34.551832 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:34.551911 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:34.553846 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:34.557169 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:34.599238 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:03:34.619117 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:34.619414 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:34.624543 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:34.624754 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:34.630753 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:34.634364 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:34.642549 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:34.709776 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:34.717983 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:34.719152 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:34.726629 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:34.728448 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:34.739471 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:34.821342 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:34.821954 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:34.832873 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:34.837234 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:34.846325 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:34.854256 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:34.862465 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:03:34.866640 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:34.874634 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:34.882130 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:34.888227 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:34.898237 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:34.900453 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:34.918694 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:34.986615 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:34.986707 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:34.987340 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:34.992422 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:35.005482 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:35.006469 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:35.060870 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:35.061690 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:35.063227 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:35.065337 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:35.070104 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:35.082597 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:35.139340 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:35.145615 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:35.145900 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:35.146205 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:35.151796 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:35.162156 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:35.162221 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:35.173295 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:35.182705 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:35.257961 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:35.266187 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:35.266227 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:35.272600 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:35.275951 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 08:03:35.276404 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:35.302104 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:35.333849 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:35.335561 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:35.336024 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:35.340037 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:35.345217 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:35.346102 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:35.366744 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:03:35.444787 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:35.448117 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:35.450253 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:35.459713 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:35.462260 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:35.463333 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:35.477828 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:35.535614 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:35.558650 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:35.563398 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:35.563864 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:35.570258 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:35.574919 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:35.587014 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:35.606523 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:35.608952 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:35.634227 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:03:35.662338 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:35.662542 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:35.675576 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:35.677798 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:35.678396 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:35.684691 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:35.715003 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:35.765987 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:35.766770 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:35.776039 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:03:35.777828 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:35.780599 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:35.782506 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:35.785228 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:35.789121 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:35.792644 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:35.807072 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:35.809419 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:35.817122 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:35.817500 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:35.819782 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:35.850030 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:35.943797 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:35.976246 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:35.983611 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:35.984007 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:35.986619 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:35.989399 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:35.990722 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:36.014571 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:36.014909 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:36.016633 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 08:03:36.017987 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:36.020663 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:36.067116 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:36.077397 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:36.077427 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:36.080940 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:36.083065 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:36.093140 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:36.095731 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:36.096402 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:36.109793 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:36.126816 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:36.236215 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:36.236204 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:36.245082 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:36.245980 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:36.246594 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:36.257651 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:36.269861 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:36.281291 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:36.281909 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:36.283581 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:36.284394 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:36.293419 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:36.296350 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:36.343410 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:36.390231 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:36.400427 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:36.439155 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:36.439953 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:36.445775 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:36.451316 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:36.461493 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:36.462193 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:36.462970 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:36.463115 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 08:03:36.464177 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:03:36.465973 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:36.470692 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:36.473311 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:03:36.474220 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:36.502812 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:36.507305 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:36.511937 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:36.513140 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:36.527354 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:36.530714 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:36.531458 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:36.534512 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:36.541747 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:36.544481 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:36.567067 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:36.582736 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:36.645181 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:03:36.672693 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:36.679472 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:36.691882 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:36.694152 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:36.700834 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:36.703490 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:03:36.713132 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:36.716065 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:36.734997 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:36.738693 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:36.739673 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:36.742120 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:36.774791 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:36.895120 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:36.898099 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:36.906155 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:36.908547 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:36.909149 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:36.932459 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:36.953575 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:36.959451 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:36.959849 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:36.960088 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:36.976589 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:03:36.978991 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:36.979423 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:36.990836 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:37.027534 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 08:03:37.044474 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:37.046689 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:03:37.077553 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:37.083608 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:37.083906 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:37.084008 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:37.105475 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:37.119668 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:37.128800 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:37.133718 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:37.142009 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:37.160389 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:37.163481 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:37.175354 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:37.177266 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:37.234710 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:37.259244 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:37.262758 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:37.273038 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:37.275677 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:37.279436 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:37.291703 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:37.342072 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:37.345335 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:37.345449 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:37.345700 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:37.347563 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:37.373744 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:37.434270 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:37.434391 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:37.434958 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:37.438045 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:37.447708 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:37.449520 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:37.460402 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:37.501911 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:03:37.506003 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:37.506269 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:37.511165 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:37.513288 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:37.513584 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:37.521828 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:37.558522 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:37.600162 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:37.621713 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 08:03:37.631197 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:37.632680 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:37.636952 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:37.645390 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:37.661660 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:37.664932 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:37.738695 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:37.743017 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:37.745338 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:37.745944 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:37.758985 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:37.783131 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:37.813073 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240501 08:03:37.813690 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:37.843026 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:37.847744 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:37.850920 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:37.854920 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:37.855310 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:37.882586 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:37.883137 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 08:03:37.883309 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:37.909955 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:37.918365 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:37.921409 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:37.922192 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:37.926409 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:37.944257 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:37.969437 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:03:38.008270 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:03:38.023849 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:38.026620 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:38.027632 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:38.027678 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:38.034782 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:38.037878 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:38.043323 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:38.071080 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:38.101511 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:38.101799 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:38.114792 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:38.116688 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:38.123067 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:38.123468 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:38.189679 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:38.216078 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:38.218572 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:38.219221 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:38.224123 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:38.227211 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:38.241688 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:38.249625 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:38.254528 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:38.255985 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:38.269578 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:38.280654 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:38.286268 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 08:03:38.286631 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:38.295877 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:38.318483 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:38.350174 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:38.361457 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:38.361908 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:38.373337 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:38.374835 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:38.384234 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:38.387573 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:38.395206 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:03:38.489871 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:38.494124 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:38.494738 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:38.497143 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:38.497310 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:38.519167 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:38.531309 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:38.612536 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:38.614257 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:38.619746 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:38.625322 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:38.629863 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:38.637390 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:03:38.639129 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:38.642917 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:38.691361 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:38.694607 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:38.711234 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:38.722472 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:38.723709 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:38.738197 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:38.741890 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:03:38.742372 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:38.743073 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:38.765239 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:38.781005 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:38.783846 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:38.792732 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:38.793124 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:38.793536 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:38.795049 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:38.887249 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:38.898699 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:38.909233 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:38.913129 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:38.922945 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:38.945876 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:38.950067 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:38.963788 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:38.967664 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:38.981150 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:38.983038 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:39.003053 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:39.008760 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:03:39.009428 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:39.027087 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:39.030104 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:39.045445 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:39.071374 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:39.075052 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:39.076093 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:39.093971 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:39.103135 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:39.104651 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:39.105423 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:39.106539 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:39.108968 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:39.173996 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:39.194254 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:39.194561 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:39.194666 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:39.199095 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:39.199115 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:39.199436 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:39.207918 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:39.217572 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:39.223085 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:39.229542 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:39.235100 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:39.245836 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:39.385428 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:39.385421 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:39.385869 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:39.387207 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:39.398612 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:39.398928 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:39.403013 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:39.411504 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:39.414181 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:39.419484 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:39.433797 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:39.434818 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:39.449734 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:39.464028 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:39.479594 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:39.530823 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:03:39.531620 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:39.534387 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:39.534488 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:39.547497 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:39.548063 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:39.549422 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:39.550119 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:39.600811 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:03:39.621227 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:39.621728 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:39.623862 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:39.629478 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:39.641940 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:39.644556 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:39.675935 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:39.706759 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:39.714143 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:39.749333 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:39.750885 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:39.757539 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:39.758147 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:39.777462 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:39.785459 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:39.856849 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:39.858245 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:39.864356 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:39.869055 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:39.871863 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:39.883185 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:39.946413 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:39.966939 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:39.975392 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:39.988360 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:39.989508 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:39.990290 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:39.995065 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:39.995779 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:40.010646 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:40.034669 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:40.034968 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:40.043402 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:40.055575 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:40.094628 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:40.098292 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:40.100162 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:40.100430 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:40.100450 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:40.103284 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:40.119261 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:40.180696 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:40.181174 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:03:40.223878 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:40.226905 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:40.227120 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:40.231019 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:40.239554 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:40.253232 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:40.355307 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:40.359360 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:40.380172 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:40.383559 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:40.397707 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:40.401618 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:03:40.402766 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:40.414927 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:03:40.422791 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:40.426834 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:40.428401 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:40.429045 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:40.439728 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:40.441541 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:40.442723 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:40.489692 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:40.504280 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:40.511467 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:40.514953 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:40.520582 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:40.532027 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:40.651584 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:40.651654 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:40.659456 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:40.661474 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:40.663614 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:40.663762 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:40.663838 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:40.676359 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:40.676714 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 08:03:40.685418 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:40.716357 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:40.718281 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:40.724078 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:40.729079 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:40.744714 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:40.784029 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 08:03:40.802394 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:40.842876 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:40.842936 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:40.847239 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:40.851060 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:40.854635 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:40.879096 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:40.902149 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:40.905862 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:40.911314 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:40.914088 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:40.920205 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:40.922732 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:40.931335 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:40.967149 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:40.969880 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:40.971796 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:41.001585 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:41.002240 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:41.003614 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:41.009956 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:41.072175 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:41.111383 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:41.114024 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:03:41.118881 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:41.131577 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:41.132879 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:41.135617 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:41.141026 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:41.141093 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:41.146989 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:41.162138 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:41.174229 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:41.174407 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:41.179368 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:41.189040 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:41.207145 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:41.341553 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:41.343091 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:41.348379 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:41.354853 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:41.355736 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:41.356473 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:41.359143 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:41.371145 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:41.377830 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:41.377920 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:41.377939 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:41.380439 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:41.383844 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:41.397231 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:41.401358 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:41.401918 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:41.417896 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:41.426398 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:41.430652 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:41.439281 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:41.446739 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:41.455462 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:41.567478 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:41.575852 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:03:41.576728 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:41.577776 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:41.578640 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:41.583822 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:41.592671 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:41.597784 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:41.623077 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:41.656265 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:41.703233 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:41.704516 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:03:41.704967 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:41.705109 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:41.707366 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:41.722551 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:41.727016 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:03:41.727747 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:41.752067 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:41.756511 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:41.839104 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:41.841457 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:41.845690 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:41.855533 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:41.858906 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:41.871079 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:03:41.879307 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:41.879648 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:41.886835 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:41.889091 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:41.892235 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:41.895236 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:41.900398 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:41.906016 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:41.923481 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:41.977905 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:03:42.031144 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:42.031518 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:42.037431 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:42.047976 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:42.053800 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:42.054594 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:42.103775 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:42.106925 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:42.111552 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:42.115147 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:42.118849 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:42.120299 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:03:42.123389 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:42.206735 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:42.213614 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:42.234040 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:42.237396 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:42.237484 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:42.250358 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:42.250950 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:42.267163 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:42.287665 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:42.304131 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:42.305123 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:42.307219 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:42.327793 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:03:42.341765 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:42.360283 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:42.361539 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:42.391068 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:42.394492 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:42.395298 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:42.399827 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:42.402683 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:42.403759 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:42.422011 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:42.476351 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:42.481304 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:42.483139 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:42.484019 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:42.492002 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:42.503301 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:42.587226 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:42.594974 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:42.595927 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:42.597651 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:42.604593 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:03:42.605361 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:42.619750 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:42.631062 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:42.671777 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:42.673203 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:42.673995 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:42.674919 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:42.676235 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:42.692595 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 08:03:42.704176 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:42.810549 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:42.820356 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:42.821187 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:42.829330 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:42.832387 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:42.832638 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:42.846452 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:03:42.848093 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:42.858016 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:42.858292 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:42.862249 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:42.871981 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:42.891072 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:42.896037 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:42.908851 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:42.966452 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:42.972429 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:42.972769 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:42.973776 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:42.991407 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:42.993575 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:42.994484 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:42.995350 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 08:03:42.996069 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 08:03:43.007225 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:43.076392 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:43.076709 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:43.080502 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:43.087049 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:43.109161 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:43.110224 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:43.111598 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:43.167554 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:43.186798 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:43.199719 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:43.200687 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:43.201572 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:43.208253 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 08:03:43.216295 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:03:43.216748 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:43.285151 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:43.289090 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:43.291014 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:43.295567 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:43.297765 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:43.306128 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:43.306272 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:43.306617 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:43.310001 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 08:03:43.314805 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:43.316332 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:43.341825 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:43.351454 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:43.352080 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:43.359604 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:43.374132 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:43.430711 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:43.466410 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:43.467175 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:43.474272 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:43.488436 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:43.506235 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:03:43.507679 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:43.509773 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:43.517648 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:43.521981 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:43.538565 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:43.539314 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:43.544185 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:43.545329 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:43.552934 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:43.555946 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:43.598187 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:03:43.676177 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:43.679983 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:43.681070 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:43.681262 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:43.682312 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:43.683219 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:43.819679 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:43.827926 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:43.831444 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:03:43.832031 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:43.832299 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:43.836426 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:43.842841 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:43.847060 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:43.899381 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:43.911211 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:03:43.921814 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:43.927631 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:43.928669 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:43.949748 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:43.949822 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:43.951119 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:44.030784 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:44.036796 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:44.043630 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:44.057626 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:44.057703 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:44.058578 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:44.064618 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:44.083599 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:44.084164 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:44.086627 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:44.102639 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:44.108599 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:44.109896 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:44.118060 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:44.126976 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:03:44.127838 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:44.172020 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:44.172031 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:44.178618 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:44.181587 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:44.184525 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:44.194751 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:44.196681 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:44.210582 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:44.258030 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:44.279824 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:44.284116 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:44.290787 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:44.291698 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:44.302974 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:44.305163 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:44.356837 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:44.357506 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:44.364444 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:44.367228 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:44.368680 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:44.376142 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:44.450744 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:44.461140 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:44.464805 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:44.468209 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:44.476650 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:44.482472 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:44.492029 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:44.511945 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:44.528156 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:44.531455 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 08:03:44.535248 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:44.538712 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:44.542068 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:44.562204 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:44.575086 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:44.585973 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:44.586118 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:44.601971 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:44.602746 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:44.602831 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:44.616650 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:44.617671 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:44.623876 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:44.629274 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:44.714948 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:44.715495 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:44.728444 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:44.734951 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:44.737902 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:44.744985 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:44.753079 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:03:44.753840 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:44.764842 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:44.798588 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:03:44.813565 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:44.821719 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:44.829757 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:44.831698 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:44.840997 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:44.841694 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:44.924513 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:44.925354 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:44.935868 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:44.942620 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:44.947322 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:44.952679 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:44.956816 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:44.974627 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:45.015549 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:45.023250 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:45.024107 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:45.024960 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:45.028612 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:45.031261 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:45.048522 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:45.052014 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:45.052769 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:45.053503 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:45.059999 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:45.079300 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:45.099896 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:45.168823 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:45.171244 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:45.185436 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:45.192024 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:45.215834 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:45.224519 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:45.239881 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:45.240985 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:45.247102 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:45.248457 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:45.254580 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:45.263959 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:45.358726 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:45.366255 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:45.384193 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:45.390955 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:45.403525 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:45.406407 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:45.456925 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:45.457302 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:45.462178 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:45.462646 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:45.469002 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:45.473233 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:03:45.492651 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:45.568639 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:45.574177 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:45.575698 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:45.576258 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:45.587605 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:45.588753 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:45.598338 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:45.605489 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:45.609596 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:45.691012 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 08:03:45.692977 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:45.701228 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:45.704463 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:45.707943 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:45.713488 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:45.717476 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:45.754297 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:45.771409 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:45.774798 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:45.782851 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:45.789922 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:45.793931 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:45.798221 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:45.798439 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 08:03:45.806735 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:45.815788 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:45.854898 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:45.857518 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:03:45.892481 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:45.892936 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:45.900961 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:45.903308 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:45.922701 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:45.926837 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:45.931777 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:45.956750 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:45.968055 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:45.968485 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:45.975759 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:45.976554 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:45.988785 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:45.994626 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:46.036442 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:46.038543 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:46.039397 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:46.042783 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:46.050683 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:46.051276 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:46.059769 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:46.090948 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:46.148049 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:46.151862 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:46.154393 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:46.162014 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:46.163766 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:46.164104 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:46.197000 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:46.223841 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:46.225634 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:46.229279 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:46.229832 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:46.245473 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:46.255059 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:46.318461 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:46.331974 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:46.338874 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:46.341442 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:46.346083 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:46.348255 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:46.375011 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:46.398787 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:46.403276 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:46.407953 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:46.408332 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:46.411314 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:46.413710 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:46.423043 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:46.468091 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:46.468936 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:03:46.473165 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:46.475939 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:46.476716 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:46.477437 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:46.484012 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:46.506502 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:46.514050 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:46.530490 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:46.558377 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:03:46.581444 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:46.601864 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:46.607913 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:46.608805 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:46.610733 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:46.613536 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:46.628991 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:46.629066 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:46.641645 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:46.651415 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:46.660384 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:46.667148 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:46.669255 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:46.779778 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:46.784993 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:46.790693 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:46.814654 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:46.820106 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:46.831435 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:46.832049 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:46.834396 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:03:46.835709 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:03:46.865747 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:46.873355 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:46.875764 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:46.877864 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:46.888958 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:46.889500 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:46.891631 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:46.905855 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:46.920182 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:46.923418 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:46.935670 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:46.947741 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:46.948279 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:46.971007 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:03:46.975658 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:47.034948 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:47.037693 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:47.041954 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:47.042631 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:47.049898 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:47.050938 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:03:47.057974 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:47.068023 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:47.079078 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:47.113512 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:03:47.160024 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:47.161828 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:47.163949 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:47.191097 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:47.192055 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:47.198899 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:47.200652 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:47.209570 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:47.220685 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:47.228749 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:47.230935 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:47.236765 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:47.238888 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:47.251435 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:03:47.312089 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:47.312215 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:47.314939 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:47.319386 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:47.320603 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:47.346930 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:47.431491 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:47.432624 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:47.433913 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:47.438117 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:47.444211 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:47.467226 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:47.481196 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:47.543941 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:47.545893 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:47.558135 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:47.562655 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:47.568253 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:47.587441 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:47.600395 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:47.623121 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:47.623173 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:47.626595 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:47.628975 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:47.629588 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:47.631484 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:47.633268 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:47.682680 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:47.683087 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:47.692442 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:47.694610 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:47.697075 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:47.719784 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:47.754760 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:03:47.755692 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:03:47.810781 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:47.817747 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:47.818535 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:47.821053 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:47.834008 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:47.834167 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:47.884651 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:47.887764 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:03:47.896191 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:47.900700 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:47.900770 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:47.902515 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:47.902788 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:47.994550 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:47.994814 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:47.995389 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:47.997139 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:48.030041 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:48.039706 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:48.055182 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:48.062459 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:48.063591 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:48.072217 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:48.080925 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:48.085520 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:48.101119 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:48.136576 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:48.137682 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:48.140506 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:48.140919 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:03:48.145920 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:48.154379 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:03:48.159690 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:48.168151 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:48.169939 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:48.170820 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:48.183946 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:48.225478 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:48.231575 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:48.234578 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:48.235565 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:48.236784 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:48.261227 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:48.268371 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:48.315567 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:48.316391 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:48.321226 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:48.331699 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:48.340034 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:48.342140 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:48.369585 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:03:48.377624 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:48.379046 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:03:48.380895 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 08:03:48.391505 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 08:03:48.400552 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:03:48.413673 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:48.413965 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:03:48.419526 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:48.424312 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:48.425303 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:48.427268 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:48.427700 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:48.442885 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:03:48.502622 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:48.504230 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:48.513985 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:48.514556 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:48.517525 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:48.548161 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:48.559720 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:48.565428 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:48.570329 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:48.575580 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:48.577833 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:48.579452 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:48.579890 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:03:48.620138 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:48.636432 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:48.694137 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:48.699530 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:48.705485 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:48.709165 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:48.725824 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:48.726140 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:48.800673 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:48.802101 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:48.810631 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:48.814375 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:48.823921 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:48.825646 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:48.841866 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:48.856513 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:03:48.857381 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:03:48.895741 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:03:48.904580 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:48.913637 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:48.914848 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:48.930379 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:48.931465 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:48.932487 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:48.994079 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:49.019263 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:49.024132 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:49.044207 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:49.044415 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:49.053644 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:03:49.063433 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:49.063452 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:49.068555 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:49.080724 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:49.081990 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:49.089435 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:49.105252 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:49.106966 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:49.111786 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:49.176690 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:49.193966 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:49.196269 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:49.197495 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:49.199561 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:49.207877 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:49.208393 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:49.261658 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:49.262476 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:49.262728 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:49.264101 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:49.270324 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:49.283061 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:49.378911 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:49.382167 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:49.387192 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:49.387606 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:49.396124 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:49.398298 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:49.406152 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:49.460489 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:03:49.469553 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:49.474064 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:49.480417 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:49.483180 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:49.495359 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:03:49.495747 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:49.498373 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:49.511999 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:03:49.515758 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:49.581913 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:49.582125 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:49.585213 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:49.591768 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:49.592003 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 08:03:49.592777 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:49.594945 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:49.595516 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:49.599179 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:03:49.609601 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:49.622473 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:49.629263 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:49.630916 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:49.632889 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:49.639760 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:49.644804 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:49.649657 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:49.739517 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:49.745004 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:49.745674 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:49.749110 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:49.761161 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:49.762923 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:49.824409 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:49.824625 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:49.836508 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:49.838037 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:49.838672 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:49.841077 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:49.846068 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:49.893545 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:03:49.894203 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:49.929276 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:49.936654 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:49.945040 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:03:49.956002 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:49.962445 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:49.972689 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:49.978448 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:50.023166 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:50.024351 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:50.027280 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:50.029078 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:50.040206 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:50.040388 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:50.042788 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:50.056645 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:50.063154 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:50.074769 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:50.091300 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:50.091607 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:50.138736 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:50.190011 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:50.190059 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:50.191061 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:50.192615 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:50.204155 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:50.204592 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:50.302677 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:50.305805 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:50.312721 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:50.313467 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:50.324955 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:50.325068 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:50.359175 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:50.364220 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:50.369938 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:50.390304 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:50.390362 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:50.403815 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:50.419365 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:03:50.426689 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:50.458354 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:50.459050 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:50.459868 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:50.466627 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:50.474678 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:50.479363 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:50.549057 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:50.555032 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:50.561818 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:50.562069 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:50.572129 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:50.572487 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:50.581357 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:50.614006 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:03:50.622560 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:03:50.630393 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:50.666595 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:50.672185 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:50.673481 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:50.684404 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:50.689167 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:50.694165 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 08:03:50.696709 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:50.758577 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:50.760620 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:50.765035 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:50.765795 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:50.780988 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:50.790398 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:50.799137 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:03:50.820013 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:50.840283 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:50.841147 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:50.850862 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:50.853251 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:50.853875 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:50.859725 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:50.863672 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:50.870229 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:50.894083 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:03:50.902326 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:03:50.915549 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:50.942533 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:50.944482 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:50.963456 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:50.966599 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:50.966827 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:50.969933 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:50.982393 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:50.983858 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:50.990340 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:50.996523 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:50.999791 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:51.010519 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 08:03:51.021667 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:51.075986 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:51.118355 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:51.119187 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:51.119693 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:51.119974 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:51.120352 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:51.120410 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:51.162395 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:03:51.214561 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:03:51.221019 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:51.222306 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:51.222615 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:51.227133 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:51.231261 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:51.234989 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:51.251641 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:03:51.308195 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:51.308878 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:51.313612 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:51.319772 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:51.321518 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:51.329659 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:51.350872 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:51.391360 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:51.391490 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:51.391633 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:51.393174 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:51.395364 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:51.398413 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:51.417181 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:51.445852 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:51.498261 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:51.503906 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:51.514465 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:51.517194 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:51.517566 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:51.517690 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:51.525815 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:51.531627 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:51.559929 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:51.561782 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:51.564922 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:51.573905 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:51.576989 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:51.583181 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:51.692019 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:51.692178 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:51.697189 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:51.722081 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:51.746495 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:51.748275 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:51.751152 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:51.755115 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:51.762324 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:51.775292 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:51.775632 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:51.776655 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:51.792529 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:51.798938 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:51.799573 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:51.800479 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:51.820251 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:51.821797 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:51.910837 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:03:51.923066 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:03:51.946556 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:51.950356 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240501 08:03:51.950780 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:51.951373 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:51.954567 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:51.954661 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:51.968297 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:51.970782 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:52.047039 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:03:52.064428 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:52.066229 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:52.067158 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:52.068559 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:52.078409 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:52.078733 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:52.093286 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:52.102641 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:52.126978 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:03:52.144740 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:52.183679 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:52.191100 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:52.192968 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:52.215276 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:03:52.215286 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:03:52.218511 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:52.232985 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:52.235062 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:52.236709 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:52.236708 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:52.264801 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:52.271117 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:52.279731 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:52.279731 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:52.281445 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:52.300400 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:52.312825 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:52.317248 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:52.317606 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:52.330348 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:52.348552 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:52.435552 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:52.439232 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:52.439546 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:52.451201 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:52.460532 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:52.468197 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:52.485674 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:52.496136 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:52.507706 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:52.514019 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:52.516891 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:52.517863 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:52.518612 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:52.518860 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:52.535251 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:52.630256 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:52.639205 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:52.648628 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:52.650889 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:52.662549 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:52.672667 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:52.672933 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:52.714012 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:52.717947 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:52.718394 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:52.722870 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:52.727423 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:52.727589 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:52.744625 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:52.764422 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:52.773087 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:52.777428 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240501 08:03:52.790128 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:52.792413 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:52.794401 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:52.889298 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:03:52.903877 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:52.906381 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:52.907433 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:52.910624 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:52.911656 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:52.938853 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:52.947114 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 08:03:52.967123 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:03:53.031746 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:53.038554 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:53.042889 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:53.048610 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:53.055267 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:53.058660 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:53.088694 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:53.092922 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:53.118252 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:53.142622 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:53.146817 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:53.147742 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:53.148972 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:53.161095 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:53.166826 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:53.202502 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:03:53.218693 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:53.218782 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:53.218977 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:53.222930 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:53.222973 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:53.228493 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:53.236786 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:53.250934 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:03:53.266711 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:53.275840 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:53.279150 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:03:53.280122 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:53.282889 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:53.282902 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:53.291031 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:53.291795 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:53.308297 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:03:53.314194 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:53.396981 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:53.397890 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:53.408077 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 08:03:53.408502 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:53.408986 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:53.409034 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:53.410460 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:53.472005 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:53.475159 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:53.483739 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:53.484012 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:53.497807 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:53.506054 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:53.541519 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:03:53.541795 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:53.543509 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 08:03:53.553231 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:53.561295 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:53.585387 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:53.585459 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:53.587914 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:53.588339 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:53.610626 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:53.622326 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:53.622535 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:53.624400 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:53.628700 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:53.643559 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:53.643587 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:53.644107 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:53.679070 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:53.727079 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:03:53.745591 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:53.746075 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:53.755955 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:53.756489 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:53.760022 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:53.777498 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:53.863783 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:53.868850 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:53.877552 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:53.880518 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:53.889899 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:53.893038 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:53.936647 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:03:53.953485 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:53.956607 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:53.957680 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:53.960970 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:53.963994 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:53.970614 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:53.984265 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:54.015249 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:54.030385 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:03:54.051431 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:54.059118 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:54.059659 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:54.063093 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:54.084762 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:54.086406 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:54.098644 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:54.101459 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:54.102599 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:54.110663 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:54.114224 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:54.117733 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:54.139539 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:54.173477 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:54.191133 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:54.200696 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:54.203554 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:54.205607 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:54.212195 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:54.297428 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:54.298777 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:54.303452 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:54.310788 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:54.321368 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:54.323513 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:03:54.337693 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:54.355120 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:54.364701 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:54.365505 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:54.382164 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:54.387077 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:54.395579 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:54.398670 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:54.456324 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:03:54.504323 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:54.508481 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:54.513072 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:54.520068 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:54.521607 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:54.553396 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:54.569226 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:03:54.570220 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:54.576740 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:54.578042 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:54.586943 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:54.587064 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:54.588933 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:54.592885 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:54.602675 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:54.604305 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:54.610757 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:54.612562 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:54.622507 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240501 08:03:54.624603 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:54.638991 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:54.643690 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:54.645823 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:03:54.645944 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:54.659202 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:54.747030 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:54.747879 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:54.752722 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:54.761448 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:54.762120 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:54.778239 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:54.846125 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:54.865650 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:54.867175 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:54.867906 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:54.871066 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:54.880734 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:54.890604 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:54.899435 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:54.905905 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:03:54.957484 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:03:55.004771 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:55.007553 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:55.018193 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:55.023458 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:55.030356 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:55.036127 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:03:55.038285 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:55.038717 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:55.043869 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:55.055488 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:55.062875 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:55.080787 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:55.084794 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:55.092682 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:55.120033 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:55.124428 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:55.125523 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:55.134513 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:55.135993 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:55.158110 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:55.158773 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:55.234753 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:55.236377 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:55.237522 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:55.238131 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:55.244642 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:55.246605 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:55.340257 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:55.353436 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:55.360278 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:55.369514 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:55.371537 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:55.375310 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:55.378662 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:55.453604 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:55.457495 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:55.469024 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:55.469897 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:55.472115 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:55.490700 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:55.504709 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:55.519963 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 08:03:55.549469 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:55.556432 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:55.569070 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:55.576251 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:55.577284 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:55.579025 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:55.585142 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:55.593348 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:55.596060 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:55.633697 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:55.633837 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:55.634717 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:55.644318 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:55.649412 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:03:55.654213 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:03:55.671972 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:55.675874 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:03:55.695044 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:55.740672 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:55.742847 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:55.750510 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:55.760890 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:55.762554 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:55.762554 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:55.768002 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:55.769799 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:55.780285 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:55.812309 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:55.814911 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:55.822614 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:55.842836 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:55.883011 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:03:55.927129 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:55.930204 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:55.934638 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:55.936733 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:55.937137 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:55.959420 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:55.997709 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:03:56.003749 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:56.004509 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:56.017503 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:56.024209 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:56.026238 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:56.036466 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:56.039283 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:56.049490 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:56.052258 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:56.055971 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:56.056777 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:56.057559 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:56.061831 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:56.102975 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:56.185837 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:56.194020 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:56.201855 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:56.216177 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:56.224298 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:56.237166 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:56.237722 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:56.251250 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:56.251562 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 08:03:56.258359 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:56.267396 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:56.273277 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:56.279592 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:56.287314 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:03:56.294653 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:56.351554 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:56.355293 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:56.360502 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:56.364459 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:56.369817 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:56.390682 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:56.428654 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:03:56.451224 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:03:56.459086 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:56.463148 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:56.467531 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:56.472333 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:56.472844 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:56.493781 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:56.497011 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:56.500718 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:56.521582 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:56.523018 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:56.528033 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:56.533716 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:56.535717 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:56.630218 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:56.631314 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:56.631772 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:56.645341 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:56.649655 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:56.654757 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:56.660213 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:03:56.665616 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:56.726320 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:56.728230 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:56.731413 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:56.738687 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:56.747208 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:56.768121 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:03:56.773650 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:56.782920 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:56.841890 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:56.844890 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:56.845937 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:56.846530 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:56.854631 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:56.857120 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:56.864764 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:56.880754 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:56.890456 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:56.892179 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:56.898144 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:56.898620 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:56.899096 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:56.911465 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:03:56.915226 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:03:56.923635 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:56.961385 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:03:57.007666 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240501 08:03:57.017554 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:57.017832 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:57.022342 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:57.024082 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:57.024586 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:57.040591 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:57.090190 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:57.119778 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:57.119778 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:57.120234 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:57.122712 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:03:57.125115 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:57.126574 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:57.146601 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:57.170099 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:57.194384 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:57.200938 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:57.202991 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:57.211701 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:57.212316 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:57.212491 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:57.285044 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:57.291208 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:57.291905 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:57.293246 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:57.293634 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:57.295723 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:57.365772 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:57.366060 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:57.367233 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:57.367667 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:57.368157 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:57.399406 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:57.451436 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:57.469954 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:57.476933 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:57.480239 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:57.481202 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:57.481942 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:57.482976 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:57.489627 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:57.489761 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:57.510952 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:57.538867 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:57.545751 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:57.552749 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:57.556720 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:03:57.564926 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:57.577898 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:57.593137 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:57.602617 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:57.611510 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:57.615453 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:57.617172 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:57.622071 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:57.627172 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:03:57.655370 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:57.722540 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:57.724499 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:57.726547 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:57.726608 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:57.731184 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:57.735438 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:57.808334 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:57.814661 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:57.815623 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:57.831678 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:57.832417 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:57.834606 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:57.880264 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:03:57.893303 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:57.915786 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:57.916042 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:57.917388 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:57.919468 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:57.923147 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:57.925284 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:57.928426 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:03:57.943161 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:03:57.958855 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:57.990198 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:57.990505 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:57.996840 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:58.004043 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:58.012797 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:58.023882 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:58.035455 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:58.044104 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:03:58.052866 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:03:58.068114 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:58.068565 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:58.075490 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:58.077195 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:58.078492 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:58.097339 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:58.125276 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 08:03:58.151330 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:58.158049 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:58.159214 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:58.160701 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:58.161149 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:58.180231 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:58.209328 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:03:58.242343 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:58.244765 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:58.247187 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:58.251124 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:58.274631 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:58.278945 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:58.327854 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:58.328379 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:58.329249 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:58.331527 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:58.332351 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:58.348131 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:03:58.350281 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:03:58.350548 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:58.376695 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:58.389391 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:03:58.413092 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:58.416798 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:58.418969 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:58.422528 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:58.427014 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:58.435065 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240501 08:03:58.451774 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:58.452584 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:58.492852 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:58.493768 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:58.494647 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:58.495150 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:58.503517 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:58.511756 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:03:58.515549 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:58.538038 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:03:58.540360 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:03:58.596436 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:58.597173 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:58.597437 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:58.604161 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:58.606622 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:58.618374 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:58.700212 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:03:58.700907 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:03:58.710822 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:58.715751 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:58.720522 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:58.725163 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:58.725353 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:58.743554 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:58.804582 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:58.815534 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:58.819623 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:58.827024 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:58.829509 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:58.844142 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:58.845363 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:58.868851 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 08:03:58.873347 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:58.906931 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:58.912097 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:58.916638 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:58.924808 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:58.940400 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:58.952798 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:58.953127 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:58.955667 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:58.968106 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:58.975437 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:03:58.983793 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:03:58.985430 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:58.987110 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:58.996191 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:59.099704 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:59.102455 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:59.109606 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:59.111531 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:59.120362 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:59.123538 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:59.145992 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:03:59.147645 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:59.153561 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:59.154554 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:59.160562 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:59.184576 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:59.185281 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:59.239969 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:59.279368 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:59.279875 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:03:59.280221 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:59.289165 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:59.290791 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:59.313891 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:59.327658 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:59.335510 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:03:59.354254 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:03:59.356225 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:59.361050 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:59.367429 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:59.367540 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:59.368825 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:59.384434 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:59.391539 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:59.392832 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:59.401605 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:59.406380 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:59.416712 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:59.420394 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:03:59.434326 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:59.444249 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:03:59.452767 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:03:59.504998 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:03:59.522698 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:03:59.543783 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:59.545337 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:59.551909 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:59.554574 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:59.562989 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:59.570600 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:59.576185 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:59.578588 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:59.578816 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:59.579124 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:59.588657 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:03:59.614171 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:03:59.619652 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:03:59.623036 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:03:59.728266 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:59.744071 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:59.745114 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:59.750614 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:59.766253 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:59.779877 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:59.788914 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:59.800000 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:59.801084 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:59.814358 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:59.824409 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:03:59.826210 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:03:59.842849 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:59.864168 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:03:59.869665 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:03:59.875787 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:03:59.876207 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240501 08:03:59.885346 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:59.892848 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:59.893571 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:03:59.972702 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:03:59.975948 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:03:59.981599 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:03:59.983121 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:03:59.993399 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:03:59.993405 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:00.013449 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:00.023995 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:04:00.037766 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:00.044265 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:00.051848 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:00.052824 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:00.059549 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:00.062580 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:00.153997 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:00.155591 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:00.158521 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:00.160455 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:00.161150 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:00.178750 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:00.180091 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:00.239724 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:00.252360 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:00.252839 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:00.253738 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:00.256805 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:00.275787 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:00.282186 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:00.350344 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:04:00.365384 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:00.365860 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:00.366078 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:00.367676 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:00.370754 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:00.371302 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:00.431389 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:00.443653 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:00.446666 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 08:04:00.451676 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:00.464519 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:04:00.495261 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:00.500977 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:00.506497 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:00.507573 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:00.515947 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:00.517282 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:00.518751 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:00.534263 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:04:00.535318 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:00.535634 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:00.536917 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:00.560997 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:00.569619 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:00.670643 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:00.672538 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:00.682439 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:00.683256 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:00.693727 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:00.708659 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:00.743654 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:04:00.747699 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:00.752380 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:00.754592 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:00.771049 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:00.777740 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:00.789155 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:00.792701 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:04:00.793218 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:00.807858 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:04:00.819554 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:00.842132 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:00.854254 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:00.856813 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:00.859710 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:00.860615 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:00.882272 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:00.954336 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:00.957438 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:00.958863 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:00.960065 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:04:00.966718 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:00.970606 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:00.975021 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:00.980119 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:04:01.050891 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:01.051414 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:01.051609 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:01.056872 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:01.064913 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:01.069923 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:01.150357 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:01.154352 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:04:01.157950 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:01.158712 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:01.160939 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:01.164902 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:01.167860 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:01.171979 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:01.182910 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:01.185726 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:01.189806 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:01.227815 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:01.232412 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:01.263924 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:01.268921 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 08:04:01.279376 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:01.287216 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:01.300850 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:04:01.303052 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:01.314201 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:01.315697 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:01.316182 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:01.316749 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:01.317493 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:01.372895 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:01.384148 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:01.384189 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:01.384500 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:01.389317 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:04:01.389734 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:01.392093 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:01.489154 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:04:01.491446 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:01.495332 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:01.506060 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:01.506089 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:01.513931 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:01.527245 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:01.572528 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:01.573179 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:01.576155 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:01.596400 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:01.604328 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:01.605815 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:01.629865 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:01.634377 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:01.644066 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:01.649317 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:01.667858 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:01.678797 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:01.699285 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:01.712874 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:01.736411 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:04:01.741228 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:01.748966 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:01.750597 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:01.752187 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:01.763716 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:01.768849 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:01.780591 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:01.816426 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:01.817494 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:01.822057 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:01.824839 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:01.830080 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:01.831319 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:01.861088 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:04:01.921504 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:01.923008 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:01.924412 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:01.926909 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:01.927671 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:01.928547 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:01.944765 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:01.995625 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:02.015878 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:04:02.017007 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:02.017583 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:02.017855 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:02.018877 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:02.019173 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:02.027067 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:02.041741 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 08:04:02.066279 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:02.072767 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:04:02.132478 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:02.135290 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:02.136898 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:02.137279 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:02.150624 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:02.150960 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:02.190876 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:02.197129 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:02.224123 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:04:02.250061 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:02.251150 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:02.260504 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:02.263100 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:02.283676 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:02.296245 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:02.296851 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:02.297518 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:02.298604 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:02.302820 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:02.320526 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:02.332258 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:02.358860 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:02.368850 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:04:02.422274 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:02.423724 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:02.425773 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:02.426470 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:02.436779 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:02.456106 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:02.498266 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:02.508198 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:02.509575 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:02.513317 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:02.515100 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:02.532488 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:02.554539 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:02.560259 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:02.565613 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:02.568490 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:02.589124 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:02.589632 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:02.682551 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:02.689054 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:02.696506 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:02.706854 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:02.712270 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:02.723906 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:02.733777 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:02.739758 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:02.741176 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:02.750505 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:02.771987 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:02.775928 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:02.785316 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:02.795586 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:02.803989 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:02.847980 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:04:02.893515 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:02.894994 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:04:02.897632 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:02.902616 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:02.909396 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:02.926360 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:04:02.945286 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:02.951367 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:02.951548 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:02.956454 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:02.970269 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:02.977205 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:02.995637 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:02.997334 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:02.998613 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:03.000456 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:03.007854 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:03.009697 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 84 similar messages]
W20240501 08:04:03.011713 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:04:03.017098 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:03.025285 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:03.028496 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:03.050740 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:03.077647 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:04:03.141330 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:03.152837 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:03.154330 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:03.154999 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:03.159292 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:03.159312 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:03.177016 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:03.177397 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:03.178354 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:03.182665 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:04:03.186570 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:03.187930 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:03.205744 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:03.206696 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:03.264441 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:03.269927 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:04:03.331965 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:03.343526 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:04:03.344930 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:03.346129 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:03.347328 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:03.350476 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:03.361995 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:03.372642 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:03.374625 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:03.390560 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:03.399055 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:03.399621 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:03.409085 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:03.431681 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:03.432650 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:03.437996 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:03.438779 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:03.454217 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:04:03.466248 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:03.466614 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:03.475752 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:04:03.482965 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:03.508782 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240501 08:04:03.555936 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:03.557410 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:03.561123 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:03.564275 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:03.565469 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:03.596717 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:03.647001 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 08:04:03.674305 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:03.682328 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:03.683347 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:03.688644 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:03.690075 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:03.699313 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:03.715169 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:03.732911 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:03.760870 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:04:03.762466 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 08:04:03.810482 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:03.813375 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:03.816076 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:03.826483 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:03.827432 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:03.851181 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:03.856082 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:03.860649 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:03.868525 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:03.874610 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:03.883246 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:03.907578 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:03.911269 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:03.988986 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:04:04.001303 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:04.001596 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:04.001673 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:04.006536 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:04.006564 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:04.025612 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:04.055758 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:04:04.065716 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:04.067626 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:04.069950 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:04.070181 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:04.079365 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:04.090799 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:04.117844 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:04.159107 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:04.184361 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:04.197481 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:04.205123 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:04.205405 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:04.206141 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:04.213443 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:04:04.214936 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:04.233605 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:04.235670 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:04.243376 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:04.251482 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:04.252002 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:04.253528 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:04.296813 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:04:04.304271 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:04.304940 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:04.310729 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:04.311363 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:04:04.312326 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:04.314584 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:04.319310 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:04.345322 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:04.407888 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:04.428629 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:04.432682 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:04.437978 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:04.438704 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:04:04.439612 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:04.443137 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:04.463871 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:04.498723 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:04.503630 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:04.506282 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:04.506914 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:04.514636 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:04.529515 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:04.587395 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:04.597452 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:04.602979 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:04.625380 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:04:04.637549 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:04.639001 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:04.659673 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:04.662801 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:04.672019 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:04:04.673074 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:04.691431 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:04.700047 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:04.700892 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:04.703797 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:04.704056 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:04.712875 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:04.718637 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:04.723636 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:04.724498 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:04.735682 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:04.748175 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:04.749890 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:04.751689 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:04.760826 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:04.809935 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:04.816164 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:04.890347 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:04.893803 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:04.901131 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:04.914230 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:04.921238 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:04.928359 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:04.934470 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:04.938028 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:04.939008 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:04.952347 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:04.958310 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:04.962162 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:05.103544 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:05.110867 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:05.111588 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:04:05.112890 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:05.113837 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:05.118722 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:05.127491 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:05.128345 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:05.131736 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:05.132113 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:05.160326 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:05.162070 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:05.172859 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 08:04:05.173043 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:05.174947 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:05.193043 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:05.197993 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:05.208673 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:05.209298 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:05.218928 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:05.254143 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:05.277891 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:05.311128 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:05.318598 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:05.321467 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:05.324110 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:05.326059 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:05.327185 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:05.337695 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:05.404618 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:04:05.440207 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:05.440495 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:05.440886 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:05.446887 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:05.448088 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:05.459157 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:05.505805 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 08:04:05.533033 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:04:05.559278 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:05.563992 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:05.564721 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:05.584192 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:05.584661 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:05.587507 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:05.594957 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:05.595629 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:05.604339 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:04:05.618642 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:05.619500 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:05.632486 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:05.640527 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:05.715694 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:05.734107 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:05.736018 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:05.743570 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:05.745272 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:05.757681 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:05.764312 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:05.766849 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:05.806855 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:05.815054 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:04:05.837963 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:05.838310 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:05.840111 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:05.841830 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:05.846596 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:05.853938 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:05.856269 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:04:05.858954 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:05.906484 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:05.947367 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:05.949112 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:04:05.954401 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:05.957139 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:05.957558 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:05.970611 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:05.995294 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:06.024171 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:06.024931 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:06.027498 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:06.030900 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:06.040875 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:06.058336 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:04:06.063619 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:06.088119 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:06.092490 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:06.096051 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:06.102967 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:06.104931 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:06.113775 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:06.130604 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:06.171653 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:06.173179 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:06.185720 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:06.190389 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:06.195075 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:06.207659 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:06.263329 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:06.263764 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:06.266956 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:06.267666 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:06.275022 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:06.283082 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:06.292222 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:06.361063 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:06.364462 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:06.365255 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:06.366515 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:06.368606 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:06.380496 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:06.381942 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:06.449304 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:06.455078 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:06.459414 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:06.460963 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:06.473220 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:06.474201 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:06.509420 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:06.517807 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:06.518944 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:06.522868 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:06.546907 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:06.551281 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:06.581851 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:06.582063 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 08:04:06.583177 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:06.586870 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:06.590049 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:06.595235 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:06.599623 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:06.603981 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:06.604871 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:06.612265 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:06.614519 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:06.695820 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:06.699486 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:06.703894 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:06.709700 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:06.710750 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:04:06.716939 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:06.727708 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:06.729044 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:06.774030 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:06.774873 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:06.787381 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:04:06.787617 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:06.788707 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:06.800168 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:06.801029 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:06.814618 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:04:06.821599 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:06.887313 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:06.893764 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:06.896736 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:06.898638 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:06.910205 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:06.944231 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:06.944247 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:06.951200 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:06.954634 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:06.957547 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:06.962364 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:06.975337 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:06.980720 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:07.008682 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:07.016292 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:07.078408 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:07.086475 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:07.086773 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:07.092474 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:07.093029 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:07.093422 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:07.149152 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:07.169992 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:04:07.171579 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:07.173606 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:07.175841 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:07.186388 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:07.190969 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:07.199641 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:07.275626 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:07.277760 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:07.283592 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:07.285223 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:07.295354 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:07.295360 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:07.302896 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:07.332973 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:04:07.344858 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:04:07.361245 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:07.375335 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:07.375792 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:07.377498 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:07.389261 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:07.414081 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:07.425398 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:07.425684 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:07.434172 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:07.435001 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:07.437799 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:07.440647 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:04:07.442327 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:07.444329 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:07.551200 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:07.552698 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:07.559208 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:07.561293 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:07.562256 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:07.575266 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:07.657425 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:07.659013 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:07.663988 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:07.667788 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:07.680680 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:07.695307 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:07.713697 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:07.741750 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 08:04:07.757659 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:07.760819 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:07.763432 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:07.774641 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:07.780987 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:07.791323 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:07.791603 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:07.849552 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:07.854795 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:07.858901 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:07.866629 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:07.877940 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:07.883049 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:07.903252 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:07.908857 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:07.915133 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:07.918210 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:07.922317 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:07.929906 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:07.932830 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:07.936470 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:07.979781 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:08.016366 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:08.016703 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:08.016953 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:08.025923 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 08:04:08.029011 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240501 08:04:08.032531 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:08.032900 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:08.060427 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:08.069329 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:08.098268 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:08.099723 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:08.103441 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:08.107314 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:08.114609 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:08.119002 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:08.169967 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:08.172973 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:04:08.174786 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:08.184440 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:08.190400 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:08.196045 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:08.204855 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:08.205866 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:08.208380 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:08.255899 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:08.281615 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:08.287597 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:08.290221 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:08.293150 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:08.297555 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:08.298514 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:08.317571 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:08.376433 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:08.380731 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:08.388095 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:08.399655 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:08.406883 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:08.411176 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:08.415082 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:08.422621 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:08.432204 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:04:08.436756 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:08.446141 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:08.458386 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:08.462401 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:08.468775 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:08.516758 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:08.520381 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:08.521620 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:08.522177 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:08.524976 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:08.526082 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:08.538887 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:08.544649 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240501 08:04:08.595734 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:08.638458 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:08.639636 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:08.641383 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:08.642181 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:08.647162 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:08.648984 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:08.694144 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:08.763684 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:04:08.764106 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:08.771658 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:08.773944 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:08.793406 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:08.814208 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:08.823077 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:08.825634 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:08.827477 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:08.827716 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:08.837596 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:08.839309 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:04:08.847375 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:08.848371 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:08.862481 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:08.902087 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:04:08.904479 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:08.907435 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:08.920675 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:08.938258 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:08.952183 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:08.960986 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 08:04:08.962059 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:09.012995 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:09.013617 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:09.017997 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:09.028842 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:09.035960 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:09.044598 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:09.079105 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:09.097188 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:09.098080 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:09.099349 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:09.100558 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:09.122984 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:09.127342 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:09.132278 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:09.136184 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:09.159019 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:09.188555 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:09.193804 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:09.194936 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:09.198880 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:09.206264 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:09.222205 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:09.223351 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:09.255836 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:09.271342 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:04:09.296432 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:09.303645 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:09.304654 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:09.334596 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:09.335773 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:09.346022 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:09.361449 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:09.367637 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:09.372351 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:09.373389 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:09.375625 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:09.382778 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:09.389824 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:09.412340 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:09.467087 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:04:09.477789 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:09.479784 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:09.492879 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:09.499675 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:09.506886 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:09.510811 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:09.546078 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:09.550010 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:09.551857 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:09.554454 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:09.571007 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:04:09.573746 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:09.579253 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:09.628852 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:09.695106 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:09.701112 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:09.701536 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:09.701902 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:09.706050 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:09.720971 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:09.723261 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:09.728365 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:09.728780 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:09.734896 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:09.737848 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:09.747535 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:09.787137 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:09.797057 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:09.806393 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:09.807113 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:09.818060 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:09.819943 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:09.843724 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:09.852430 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:09.882673 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:09.887102 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:09.940030 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:09.942767 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:09.948758 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:04:09.951402 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:09.954454 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:09.955508 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:09.966965 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:10.009243 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:10.011574 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:10.013121 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:10.016379 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:10.016660 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:10.026269 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:10.120483 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:10.131805 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:10.132228 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:10.138864 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:10.150127 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:10.153247 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:10.183347 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:10.193876 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:10.196663 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:10.198798 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:10.201958 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:10.204667 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:10.209245 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:10.227943 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 08:04:10.228612 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:10.246335 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:10.315088 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:10.315924 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:10.321475 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:10.326630 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:10.330909 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:10.334712 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:10.354597 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:10.365459 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:10.380239 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:10.387157 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:10.395089 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:10.400411 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:10.409289 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:10.419266 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:10.431843 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:10.498703 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:10.503842 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:10.504431 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:10.505005 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:10.530737 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:10.532545 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:10.570093 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:10.570132 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:10.602161 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:10.606635 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:10.608307 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:10.612337 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:10.617921 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:10.635099 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:10.639883 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:10.645998 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:10.651615 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:10.656303 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:10.658367 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:10.677553 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:10.680342 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:10.702737 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:10.707051 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:10.730142 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:10.742986 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:10.744035 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:10.746899 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:10.748593 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:10.758514 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:10.760778 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:10.858350 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:10.858929 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:10.876813 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:10.877493 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:10.879176 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:10.880110 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:10.961088 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:10.964759 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:10.966356 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:10.987488 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:10.988561 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:04:10.988982 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:11.002256 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:11.026878 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:11.046823 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:11.055233 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:11.063485 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:11.066972 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:11.073124 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:11.073596 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:11.079703 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 08:04:11.081972 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:11.090605 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:11.103056 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:04:11.105616 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:11.106133 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:11.107791 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:11.116057 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:11.126358 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:11.137725 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:11.205942 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:11.207450 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:11.210119 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:11.212416 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:11.223515 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:11.223582 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:11.225666 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:11.233336 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:11.316284 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:11.316391 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:11.316433 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:11.316761 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:11.317030 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:11.321134 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:11.456743 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:11.460036 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:11.460912 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:11.467419 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:11.478580 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:11.497296 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:11.500166 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:11.513206 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:11.517675 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:11.539077 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:11.548044 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:11.550120 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:04:11.551138 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:11.555820 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:11.572666 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:11.575985 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:11.586658 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:11.593528 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:11.605119 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:11.612758 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:11.623365 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:11.639901 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 08:04:11.660876 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:04:11.689002 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:11.690307 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:11.691684 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:11.692001 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:11.692509 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:11.695305 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:11.723129 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:11.724601 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:11.742669 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:11.809065 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:11.810621 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:11.813046 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:11.821461 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:11.822732 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:11.823324 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:11.835902 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:11.852746 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:11.943286 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:11.952777 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:11.957252 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:11.957571 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:11.960865 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:11.978991 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:11.982612 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:11.991462 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:04:12.004142 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:12.015563 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:12.024191 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:12.026525 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:12.029048 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:12.030635 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:12.031208 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:12.038412 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:12.041241 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:12.043569 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:12.067523 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:12.089397 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:12.090581 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:12.158329 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:12.174173 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:12.179859 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:12.183140 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:12.188408 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:12.190039 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:12.203014 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:12.211074 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:12.231763 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:12.241210 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:12.249251 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:12.249897 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:12.254761 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:12.256080 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:04:12.256258 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:12.324124 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:12.360883 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:12.378671 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:12.384528 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:12.390033 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:12.397069 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:12.399314 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:12.404213 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:04:12.404506 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:12.404995 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:12.410243 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:12.425510 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:12.425510 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:12.433552 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:12.446242 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:12.446589 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:12.457854 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:12.504060 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:12.506242 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:12.511431 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:12.514269 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:12.524900 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:12.538096 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:12.550760 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:12.630651 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:12.644389 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:12.644543 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:12.646072 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:12.653326 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:12.653985 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:12.654664 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:12.655133 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:12.668627 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:12.681928 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:12.689342 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:12.707269 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:12.725184 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:12.807965 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:12.815311 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:12.822369 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:12.824563 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:12.825433 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:12.832139 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:12.847527 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:12.879381 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:12.884192 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:12.901618 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:12.904341 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:12.905754 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:04:12.910964 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:12.919209 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:12.944222 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:12.947081 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:12.980265 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:12.983790 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:12.984352 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:12.989707 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:13.002879 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:13.009562 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:13.030011 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 08:04:13.051692 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:13.058482 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:13.058714 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:13.058780 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:13.059520 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:13.061066 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:13.083907 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:13.138826 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:04:13.164579 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:13.166213 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:13.170157 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:13.184423 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:13.185473 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:13.197430 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:13.206348 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240501 08:04:13.274667 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:13.276114 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:13.276747 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:13.277951 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:13.295066 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:13.310886 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:13.338413 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:13.353962 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:13.358410 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:13.370337 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:13.369190 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:13.371158 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:13.384019 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:13.389402 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:13.393079 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:13.407438 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:13.409885 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 08:04:13.417765 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:13.476776 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:13.476799 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:13.486296 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:04:13.486827 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:13.488974 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:13.501444 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:13.502599 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:13.517418 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:04:13.554816 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:13.557226 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:13.559017 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:13.565085 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:13.569605 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:13.590292 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:13.592983 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:04:13.593681 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:13.606257 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:13.674683 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:13.682188 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:13.688122 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:13.691710 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:13.691726 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:13.714846 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:13.748538 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:13.753755 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:13.754109 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:13.754598 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:13.755439 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:13.759748 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:13.798703 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:13.807361 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:13.834017 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:13.838485 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:13.846035 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:13.846666 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:13.851439 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:13.857748 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:13.866492 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:13.899717 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:13.908773 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:13.914845 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:13.927611 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:13.937873 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:13.945808 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:14.010093 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:14.010666 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:14.021517 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:14.035842 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:14.038522 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:04:14.038683 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:14.047425 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:14.058116 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 08:04:14.060173 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:14.061947 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:14.078114 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:14.081046 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:14.085148 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:14.086493 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:14.088554 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:14.098150 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:14.185681 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:14.232477 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:14.236701 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:14.236860 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:14.238739 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:14.241554 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:14.250856 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:14.255091 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:14.255111 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:14.268201 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:14.275597 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:14.277916 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:14.284345 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:14.293025 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:14.339347 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:14.350600 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:04:14.353973 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:14.356375 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:14.357357 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:14.360128 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:14.360657 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:14.377903 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:14.407825 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:14.449223 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:14.451686 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:14.456174 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:14.460451 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:14.463330 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:14.493592 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:04:14.497599 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:14.505748 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:14.505828 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:14.510109 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:14.520076 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:14.522702 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:04:14.537585 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:14.537740 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:14.571534 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:14.596884 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:14.605307 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:14.620730 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:14.634352 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:14.634724 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:14.638669 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:14.641728 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:14.643076 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:14.664673 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:14.666868 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:04:14.694396 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:14.698716 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:14.702471 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:14.708223 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:14.741498 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:14.751271 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:14.767092 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:14.791371 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:14.803308 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:14.805614 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:14.810216 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:14.820257 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:14.822103 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:14.836884 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:14.883632 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:14.884202 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:14.885725 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:14.890589 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:14.907083 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:14.911113 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:14.913262 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:15.023882 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:15.059859 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:15.061709 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:15.062299 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:15.062568 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:15.081717 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:15.082324 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:15.138248 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:15.167260 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:15.170724 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:15.177242 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:15.182852 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:15.188089 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:15.216195 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:15.222833 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:15.248276 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:15.251529 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:15.258332 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:15.261118 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:15.263751 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:15.277987 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:15.308971 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240501 08:04:15.321025 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:15.321478 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:15.328394 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:15.328642 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:15.332206 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:15.344271 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:15.347362 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:15.362927 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:15.404520 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:15.423205 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:15.428469 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:15.433144 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:15.433279 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:15.433898 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:15.436131 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:15.437826 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:15.485076 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:04:15.505559 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:15.507844 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:15.510641 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:15.518982 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:15.552421 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:15.553303 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:15.588166 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:15.602684 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:15.609126 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:15.625021 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:15.644654 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:15.646085 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:15.646996 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:15.651630 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:15.656039 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:15.666112 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:15.666657 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:15.670718 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:15.675122 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:15.712456 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:15.760293 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:15.768034 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:15.768528 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:04:15.768667 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:15.774585 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:15.775084 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:15.787333 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:15.806380 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:15.861163 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:15.867444 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:15.869777 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:15.879213 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:15.879590 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:15.896584 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:15.958676 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:15.958796 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:15.959350 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:15.968334 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:15.974972 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:15.986541 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 08:04:15.992115 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:04:16.005231 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:16.033987 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:16.064059 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:16.070622 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:16.075385 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:16.081198 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:16.102515 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:16.102640 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:16.127905 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:16.130136 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:16.131137 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:16.136308 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:16.148593 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:16.166857 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:16.169754 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:16.170619 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:16.204063 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:04:16.204113 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:16.213728 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:16.216202 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:16.216897 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:16.225744 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:16.225931 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:16.233790 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:04:16.245054 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:16.302002 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:16.305209 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:16.311810 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:16.316450 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:16.322180 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:16.329435 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:16.379781 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:16.382109 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:16.384411 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:16.385968 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:16.388955 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:16.390516 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:16.401104 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:16.427695 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:16.440219 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 08:04:16.486196 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:16.488299 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:16.492760 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:16.495697 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:16.513135 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:16.523592 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:16.570879 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:16.577580 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:16.585547 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:16.586369 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:16.600142 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:16.603509 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:16.607051 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:16.612922 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:16.624325 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:16.627123 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:16.639264 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:16.640923 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:16.641799 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:16.756800 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:16.764513 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:16.772876 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:16.779927 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:04:16.826647 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:16.826929 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:16.828135 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:16.831807 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:16.832337 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:16.836442 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:16.869670 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:16.876359 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:16.897365 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:16.951596 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:16.961410 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:04:16.970717 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:16.979774 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:16.980656 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:16.981066 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:04:16.989037 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:16.990590 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:17.002401 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:17.005466 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:17.024971 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:17.028544 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:17.031605 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:17.050673 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:17.060442 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:17.064026 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:17.071740 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:17.075405 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:17.080655 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:17.099221 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:17.103946 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:17.105207 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:17.192232 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:17.211380 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:04:17.215415 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:17.219992 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:17.220849 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:17.224746 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:17.224916 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:17.228977 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:17.231977 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:04:17.235291 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:17.250758 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:17.255416 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:17.266193 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:17.270496 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:17.279403 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:17.407052 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:04:17.409828 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:17.411430 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:17.418785 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:17.424710 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:17.426976 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:17.428454 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:17.434636 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:17.442727 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:17.445847 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:17.452491 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:17.473374 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:17.493484 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:17.496049 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:17.497776 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:17.504954 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:17.507447 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:17.508209 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:17.511324 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:17.517745 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:17.550351 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:17.626930 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:17.629163 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:17.633668 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:17.638231 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:17.638757 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:17.659315 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:17.674729 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:17.682503 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:17.684674 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:17.688949 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:17.700637 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:17.721588 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:17.809307 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:17.815225 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:04:17.826134 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:17.828364 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:17.830456 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:17.835855 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:17.846720 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:17.883977 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:04:17.891722 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:17.892058 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:17.893193 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 08:04:17.903537 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:17.914817 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:17.915843 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:04:17.919901 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:17.923520 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:17.949880 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:17.950280 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:17.959159 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:17.966284 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:17.966672 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:17.971139 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:17.985558 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:17.993268 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:18.011700 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:18.064002 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240501 08:04:18.067526 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:18.067973 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:18.077322 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:18.080264 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:18.082845 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:18.094916 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:18.130241 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:18.137665 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:18.137856 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:18.138271 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:18.142465 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:18.149049 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:18.212422 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:04:18.213120 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:18.219451 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:04:18.249943 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:18.256705 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:18.260350 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:18.267809 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:18.275146 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:18.295897 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:18.304939 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:04:18.327811 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:18.329460 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:18.337195 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:18.351934 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:18.356595 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:18.359205 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:18.388904 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:18.396930 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:18.397540 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:18.400346 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:18.407047 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:18.415632 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:18.419595 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:18.434091 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:18.470063 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:18.505795 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:18.557216 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:18.570782 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:18.598268 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 08:04:18.615027 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:18.620460 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:18.620738 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:18.621337 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:18.622378 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:18.634841 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:18.664984 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:04:18.743077 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:18.745695 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:18.756963 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:18.763803 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:18.767025 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:18.768079 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:18.784123 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:04:18.795821 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:18.827932 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:18.828615 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:18.834372 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:18.837186 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:04:18.838725 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:18.847368 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:18.848769 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:18.863408 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:18.874238 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:18.882900 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:18.893643 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:18.899515 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:18.903357 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:18.918917 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:18.994732 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:19.001917 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:19.001912 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:19.010443 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:19.011943 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:19.031150 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:19.052327 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:19.056617 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:19.057787 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:19.069420 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:19.081801 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:19.086197 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:19.092128 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:19.166019 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:19.178787 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:19.187114 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:19.187368 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:04:19.193102 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:19.194357 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:19.198591 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:19.200830 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:19.250058 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:04:19.252197 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 08:04:19.265096 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:19.268807 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:19.274758 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:19.283078 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:19.297058 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:19.306265 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:19.308377 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:19.310170 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:19.316587 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:19.324028 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:04:19.324795 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:19.326565 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:19.327131 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:19.334314 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:19.335541 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:19.336199 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:19.349159 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:19.349272 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:19.369913 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:19.394302 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:04:19.408979 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:19.416352 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:19.417883 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:19.419737 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:19.429467 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:19.431309 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:19.484614 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 08:04:19.487092 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:04:19.507004 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:19.508761 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:19.509233 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:19.522799 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:19.525498 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:19.551931 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:19.567297 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:19.576038 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:19.584192 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:19.585189 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:19.585927 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:19.599660 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:19.636262 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:04:19.689524 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:19.695072 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:19.712271 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:19.718544 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:19.722688 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:19.724622 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:19.728814 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:19.740761 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:19.741276 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:19.742276 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:19.745605 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:19.753520 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:19.754758 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:19.792259 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:19.793409 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:19.814199 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:19.814813 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:19.814965 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:19.815616 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:19.816767 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:19.965615 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:19.965893 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:19.972128 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:19.977227 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:19.981328 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:19.982147 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:19.983958 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:19.990531 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:19.998667 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:20.020151 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:20.025568 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:20.031347 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:20.142154 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:20.143038 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:20.148661 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:20.153331 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:20.159070 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:20.161759 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:20.165558 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:20.189303 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:20.190596 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:20.195928 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:04:20.206051 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:20.206046 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:20.208189 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:20.291481 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:20.293082 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:20.324905 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:20.327836 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:20.329408 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:20.329715 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:20.333495 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:20.337111 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:04:20.340384 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:20.405169 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:20.408018 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 08:04:20.433645 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:20.436161 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:20.436580 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:20.436847 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:20.440457 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:20.458441 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:20.531112 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:04:20.548321 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:20.560992 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:20.565680 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:20.567577 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:20.568161 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:04:20.571696 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:20.572842 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:20.605352 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:20.629753 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:04:20.635882 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:20.635906 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:20.636004 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:20.636418 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:20.636581 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:20.693885 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:20.698268 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:20.698400 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:20.702309 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:20.702797 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:20.703997 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:20.777964 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:20.780642 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:20.790892 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:20.795255 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:20.798406 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:20.800446 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:20.800662 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:04:20.802661 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:20.804589 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:20.805622 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:04:20.819046 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:20.890477 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:20.892623 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:20.894904 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:20.896664 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:20.913681 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:20.913734 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:20.915594 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:20.950915 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:04:21.005949 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:21.008019 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:21.013214 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:21.015311 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:21.030313 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:21.034963 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:21.076944 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:21.081434 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:21.085826 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:21.089409 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:21.089545 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:21.119572 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:21.124001 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:04:21.129259 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:21.142845 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:21.147610 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:21.149806 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:21.153798 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:04:21.169875 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:21.169998 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:21.231516 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:04:21.237259 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:21.237908 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:21.247345 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:21.256407 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:21.260519 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:21.275220 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:21.275338 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:04:21.282145 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:21.325067 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:04:21.339481 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:21.342518 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:21.343770 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:21.346244 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:21.349884 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:21.364614 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:21.422996 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:04:21.443363 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:21.449187 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:21.454679 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:21.456418 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:21.462666 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:21.463589 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:21.476482 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:21.534780 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:21.540936 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:21.548971 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:21.549163 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:21.558678 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:21.559363 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:21.562480 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:04:21.563418 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:21.566341 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:21.580484 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:21.587584 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:21.598078 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:21.612730 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:21.684587 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:21.685195 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:21.691057 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:21.692101 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:21.696528 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:21.710609 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:21.749953 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:21.765467 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:21.798110 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:04:21.809703 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:21.812479 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:21.812938 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:21.815296 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:04:21.816654 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:21.817998 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:21.822594 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:21.834966 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:21.835897 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:21.846812 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:21.871917 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:21.936724 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:04:21.948205 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:21.953492 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:21.953917 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:21.963411 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:21.974009 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:21.975338 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 08:04:21.984166 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:04:21.985828 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:22.010509 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:22.012132 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:22.013173 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:22.019922 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:22.025089 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:22.026741 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:22.029362 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:22.031291 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:22.032761 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:22.050014 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:22.054677 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:04:22.057523 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:22.062793 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:22.093391 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:04:22.164067 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:22.166520 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:22.167172 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:22.174624 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:22.184515 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:22.185217 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:22.237488 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:22.248966 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:22.292404 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:22.294178 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:22.297449 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:22.302127 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:22.308804 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:22.319164 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:22.369305 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:22.369626 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:22.375329 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:22.377936 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:22.387297 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:22.408447 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:22.473065 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:22.476275 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:22.482677 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:22.486915 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:22.487785 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:22.494139 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:22.503062 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:22.504180 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:22.517760 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:22.527896 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:22.531508 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:22.541270 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:22.555734 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:22.635290 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:04:22.642848 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:04:22.646803 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:22.647663 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:22.652309 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:22.654778 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:22.662334 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:22.685146 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:22.696089 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:22.729190 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:22.733597 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:22.741737 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:22.742877 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:22.759747 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:22.764256 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:22.836540 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:22.846729 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:22.852854 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:22.853366 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:22.861379 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:22.862249 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:22.866683 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:22.891041 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:22.895201 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:22.902047 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:04:22.903846 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:22.913811 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:22.920471 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:22.929875 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:22.990536 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:22.991930 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:22.999070 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:23.000298 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:23.001780 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:23.003257 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:23.017886 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:04:23.081315 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:23.084115 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:23.084746 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:23.085515 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:23.094904 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:23.099516 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:23.112126 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:23.114900 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:04:23.115648 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:23.131170 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:23.151654 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:23.152451 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:23.160619 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:23.163872 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:23.164636 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:23.166673 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:23.185246 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:23.188349 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:23.232959 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:23.267105 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:23.268683 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:23.270243 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:23.270406 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:23.282940 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:23.291006 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:23.328109 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:23.335399 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:23.336098 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:23.339201 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:23.343889 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:23.349301 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:23.453202 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:23.458196 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:23.461083 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:23.467963 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:23.470947 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:23.472038 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:23.483286 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:23.499532 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 08:04:23.503022 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:23.513240 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:23.525399 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:04:23.527276 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:23.528050 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:23.532601 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:23.533695 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:23.534606 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:23.548084 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 08:04:23.548610 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:23.600452 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:23.603739 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 08:04:23.637375 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:04:23.667984 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:23.673861 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:23.677609 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:23.677986 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:23.683019 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:23.697876 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:23.705924 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:23.707350 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:23.711086 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:23.727356 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:23.732820 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:23.742158 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:23.742400 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:23.754006 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:23.776996 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:04:23.787253 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:23.790375 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:23.799638 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:23.808132 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:23.810115 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:23.811200 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:23.837939 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:04:23.913292 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:23.920826 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:23.923921 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:23.932965 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:23.933617 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:23.942307 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:04:23.962831 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:23.986802 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:23.987695 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:23.989898 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:23.997807 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:23.998495 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:24.014504 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:24.031113 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:24.069623 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:24.080547 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:04:24.125360 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:24.128324 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:24.130182 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:24.138479 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:24.148053 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:24.151255 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:24.152391 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:24.160871 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:24.164191 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:24.170914 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:24.174643 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:24.198236 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:24.309455 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:24.310806 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:24.311830 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:24.312811 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:04:24.315845 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:24.320084 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:24.323230 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:24.329537 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:24.338394 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:24.390652 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:24.399186 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:24.399274 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:24.409000 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:24.409291 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:24.410960 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:24.497738 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:24.498627 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:24.499923 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:24.500283 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:04:24.503111 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:24.505781 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:24.527364 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:24.527364 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:04:24.537928 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:24.550369 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:24.563133 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:24.568331 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:04:24.570367 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:24.572400 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:24.588827 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:24.591060 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:24.600189 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:24.633730 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:24.652096 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:24.652660 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:24.662402 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:24.662451 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:24.663947 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:24.670132 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:24.671738 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:24.737773 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:24.740761 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:24.744673 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:04:24.745360 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 08:04:24.751529 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:24.752501 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:24.767202 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:24.768699 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:24.802652 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:24.831679 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:24.837635 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:24.838590 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:24.841475 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:24.849362 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:24.859771 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:24.924113 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:24.930886 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:24.935600 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:24.937098 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:24.946882 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:24.947707 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:25.011274 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:25.016819 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:25.018867 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:25.019121 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:25.035565 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:25.048990 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:25.065196 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:25.094367 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:25.115650 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:04:25.126782 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:25.131574 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:25.143453 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:25.148468 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:25.150138 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:25.177559 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:25.190683 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:25.197209 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:25.208186 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:25.209226 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:25.210736 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:25.220674 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:25.226900 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:25.300551 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:25.327291 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:25.336937 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:25.347864 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:25.348629 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:25.366439 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:25.367313 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:25.384094 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:25.388994 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:25.398290 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:25.407827 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:25.414206 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:25.423856 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:25.424252 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:25.424630 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:25.424893 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:25.465795 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:25.469368 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:25.471194 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:25.506457 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:25.582648 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:04:25.596591 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:25.598193 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:25.600978 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:04:25.603735 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:25.612534 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:25.622902 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:25.624135 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:04:25.624548 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:25.637706 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:04:25.639186 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:25.639281 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:25.640771 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:25.644093 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:25.653980 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:25.661892 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:25.683221 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:25.698411 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:25.716429 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:25.817241 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:25.817737 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:25.824759 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:25.829560 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:25.835058 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:25.841567 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:25.845016 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:25.857329 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:25.860121 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:25.868607 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:25.886605 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:25.890731 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:25.900820 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:25.906688 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:25.907228 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:25.920555 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:25.921660 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:04:25.929481 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:25.935381 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:25.936947 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:25.944859 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:25.952131 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 08:04:26.050608 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:26.052611 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:26.053040 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:26.053776 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:26.054714 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:26.075234 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:26.094794 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:26.109666 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:04:26.166038 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:26.168557 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:26.169728 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:26.181564 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:26.184084 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:26.184343 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:26.219256 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:26.221217 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:26.222827 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:26.246374 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:04:26.300598 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:26.301555 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:26.308707 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:26.309403 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:26.309918 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:26.328178 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:26.339833 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:26.352301 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:26.356292 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:26.359344 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:26.365422 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:26.378569 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:26.405603 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:26.417411 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:26.444357 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:26.446341 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:26.446591 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:26.450814 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:26.463240 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:04:26.463321 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:26.468348 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:26.513563 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:26.518492 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:26.521224 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:26.523192 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:26.526088 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:26.526978 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:26.527817 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:26.633421 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:26.639019 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:26.642619 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:26.644779 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:04:26.658535 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:26.658715 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:26.661195 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:26.722503 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:04:26.740170 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:26.742988 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:26.748232 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:26.748960 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:26.759173 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:26.762269 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:26.818976 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:26.828007 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:04:26.850318 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:26.850891 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:26.854116 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:26.859103 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:26.864670 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:26.875763 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:26.900954 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:26.905217 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:26.906585 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:26.912743 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:26.927488 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:26.947139 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:27.027380 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:27.045679 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:27.047995 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:27.053340 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:27.066213 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:27.067988 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:27.071170 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:27.074088 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:04:27.095536 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:27.098708 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:27.110634 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:27.113552 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:27.115376 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:27.116465 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:27.119024 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:27.164369 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:27.167484 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:04:27.199254 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:27.200126 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:27.203871 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:27.215328 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:27.226804 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:27.227891 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:27.236769 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 08:04:27.245374 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:04:27.297837 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:27.300921 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:27.302913 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:27.309250 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:27.311298 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:27.334969 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:04:27.337460 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:27.339408 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:27.347712 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:27.362526 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:27.362941 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:27.376730 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:27.377897 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:27.505172 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:27.509162 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:27.509478 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:27.511103 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:27.514027 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:27.518852 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:27.524928 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:27.536579 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:27.540021 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:27.560768 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:27.564236 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:27.567792 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:27.568341 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:27.603554 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:27.604033 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 08:04:27.615020 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:27.615190 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:27.653956 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:27.655444 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:27.662838 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:27.663645 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:27.672448 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:27.685534 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:27.765105 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:27.774485 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:27.775447 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:27.785331 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:27.785877 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:27.785949 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:27.807868 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:27.825803 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:27.827584 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:27.830745 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:27.835146 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:27.839339 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:27.841903 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:27.938037 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:27.941406 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:27.944772 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:27.947118 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:27.952497 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:27.959574 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:27.966827 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:28.026656 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:28.036981 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:28.038142 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:28.039976 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:28.048107 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:28.051602 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:28.078490 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 08:04:28.120409 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:28.126578 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:28.129165 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 08:04:28.130324 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:28.130348 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:28.130483 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:04:28.132184 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:28.134111 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:28.141439 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:28.141410 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:04:28.143318 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:28.159904 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:04:28.206110 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:28.207727 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:28.208904 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:28.213433 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:28.213564 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:28.227586 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:28.306478 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:28.330189 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:28.331683 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:28.338038 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:28.338325 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:28.340462 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:04:28.353574 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:28.366159 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:28.368453 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:28.398895 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:28.402313 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:28.414511 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:28.418460 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:28.426800 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:28.429066 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:28.462908 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:04:28.504667 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:28.507782 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:28.511790 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:28.517772 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:28.519184 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:28.519796 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:28.547353 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240501 08:04:28.555488 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:28.564363 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:28.567425 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:28.568516 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:28.569854 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:28.586997 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:28.602259 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:28.644549 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:28.675029 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:28.675535 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:04:28.680044 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:28.680968 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:28.692087 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 08:04:28.696875 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:28.698985 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:28.701220 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:28.713246 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:28.724417 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:04:28.735357 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:28.735484 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:28.744246 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:28.752744 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:28.756266 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:28.757525 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:28.779809 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:28.842634 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:28.880267 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:28.885207 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:28.900592 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:28.904103 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:28.904127 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:28.908622 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:28.917661 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:28.918038 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:28.918296 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:28.918474 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:28.948071 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:28.951895 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:28.955369 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:04:28.992982 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:28.997033 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:28.997428 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:29.007719 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:29.014642 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:29.022234 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:29.027958 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:29.036656 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:29.102654 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:29.116413 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:04:29.122594 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:29.125305 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:29.130324 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:29.134657 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:29.142971 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:29.144361 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:29.185236 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:29.187302 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:29.188650 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:29.192183 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:29.206609 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:29.206907 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:29.234959 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:29.252990 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 08:04:29.279289 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:29.326825 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:29.327538 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:29.330129 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:29.339681 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:29.351372 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:29.356608 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:29.360702 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:29.364094 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:29.375685 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:29.381011 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:29.383452 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:29.386340 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:29.391534 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:29.415892 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:29.443868 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:29.444993 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:29.446130 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:29.447866 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:29.448755 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:29.452950 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:29.533119 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:29.540803 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:29.544395 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:29.549988 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:29.551321 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:29.553925 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:29.611394 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:29.612612 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:29.614177 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:29.614542 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:29.615543 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:29.616514 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:29.705492 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:29.709744 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:29.711457 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:29.712252 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:29.725427 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:04:29.729118 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:29.737484 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:29.770725 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:29.809904 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:29.811730 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:29.811877 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:29.816452 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:29.817138 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:29.818996 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:29.833011 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:29.833436 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:04:29.833492 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:29.843164 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 08:04:29.874809 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:29.929921 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:04:29.930464 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:29.930671 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:29.931227 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:29.941068 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:29.945978 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:29.965341 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:30.052870 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:30.062032 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:30.062588 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:30.064046 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:30.066748 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:30.075131 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:30.109340 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:30.114563 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:30.114744 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:30.116384 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:30.123310 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:30.144752 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:30.150115 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:30.166577 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:04:30.240553 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:30.242894 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:30.243113 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:30.244623 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:30.251377 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:30.252990 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:30.264489 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:30.264945 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:30.267541 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:30.277809 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:04:30.294291 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:30.295547 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:30.305227 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:30.313177 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:30.342253 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:30.368983 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:30.371166 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:30.376055 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:30.382337 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:30.382787 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:30.391196 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:30.394464 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:30.457484 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:30.460630 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:30.466986 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:30.468371 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:30.468839 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:30.481802 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:30.508049 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 08:04:30.537477 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:30.538672 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:30.541522 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:30.543156 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:30.553457 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:30.556519 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:30.569978 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:30.645992 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:30.653297 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:30.654624 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:30.654949 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:30.659943 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:30.673170 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:30.705374 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:30.706894 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:30.721591 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 08:04:30.722316 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:30.726532 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:30.732841 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:30.742307 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:30.749683 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:04:30.750286 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:30.752647 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:04:30.785851 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:30.793682 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:30.804314 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:30.807186 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:30.815999 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:30.829846 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:30.830582 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:30.832054 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:30.867065 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:30.870499 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:30.877346 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:30.879457 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:30.906989 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:30.907560 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:31.002115 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:04:31.015981 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:31.021183 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:31.023418 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:31.029109 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:31.037165 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:31.038372 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:31.050397 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:31.054643 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:31.059530 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:04:31.064713 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:31.066159 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:04:31.071568 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:31.072502 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:31.075678 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:04:31.095139 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:31.139878 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:31.159305 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:31.161816 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:31.163398 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:31.170969 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:31.172387 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:31.194559 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:31.194922 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:31.205869 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:31.214713 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:31.262221 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:31.265216 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:31.267181 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:31.271478 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:31.276446 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:31.279659 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:31.287374 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:31.293160 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:31.299839 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:04:31.313580 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:31.317487 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:31.335181 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:31.338573 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:31.340828 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:31.441985 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:31.442133 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:31.443846 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:31.450266 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:31.452414 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:31.466310 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:31.479884 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:31.530223 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:31.542973 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:31.544317 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:31.544600 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:31.549329 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:31.560937 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:31.571386 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:31.622470 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:31.643859 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:31.646030 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:31.646289 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:31.648241 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:31.652345 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:31.665552 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:31.681183 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:31.721375 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:31.744328 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:31.764823 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:31.766094 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:31.766564 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:31.792981 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 08:04:31.802182 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:31.802469 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:31.807021 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:31.829418 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:31.829571 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:31.835850 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:31.842017 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:31.848263 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:31.862303 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:31.863185 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:31.879653 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:04:31.956638 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:31.956655 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:31.964756 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:31.971184 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:31.989892 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:31.992662 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:32.054502 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:32.063647 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:32.064628 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:32.064924 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:32.069430 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:32.071549 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:32.074826 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:32.075946 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:32.083636 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:32.086045 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:32.091225 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:32.111541 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:32.123827 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:32.151540 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:32.209909 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:32.238188 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:32.253047 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:32.253060 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:32.254551 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:32.260872 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:32.273217 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:32.279785 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:32.283396 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:32.306053 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:32.317615 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:04:32.330296 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:32.340483 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:32.359807 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:32.395543 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:32.397454 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:32.402726 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:32.412211 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:32.415632 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:32.418565 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:32.434625 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:32.463176 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:32.473075 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:32.473595 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:32.476155 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:32.497905 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:32.499424 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:32.509984 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:32.553561 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:32.559242 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:32.560122 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:32.561189 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:32.561445 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:32.591914 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:32.607298 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:32.611889 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:32.655084 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:32.658077 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:32.658089 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:32.661427 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:32.663362 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:32.676699 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:32.679869 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:32.697547 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:04:32.731251 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:32.741959 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:32.744028 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:32.751860 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:32.753290 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:04:32.760987 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:32.771634 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:32.790715 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 08:04:32.817235 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:32.859954 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:32.864434 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:32.875254 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:32.879151 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:32.880632 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:32.882992 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:32.884696 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:04:32.886274 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:32.888104 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:32.894933 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:32.912869 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:32.914096 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:32.916513 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:32.923575 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:33.013522 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:33.015256 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:33.022483 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:33.034960 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:33.035538 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:33.050738 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:33.093482 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:33.096879 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:33.097745 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:33.097774 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:33.098012 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:33.106674 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:33.122713 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:33.155351 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 08:04:33.216853 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:33.217012 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:33.222057 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:33.237015 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:33.237987 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:33.239058 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:33.239091 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:33.248057 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:33.262849 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:33.284528 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:04:33.317415 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:33.322293 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:33.325961 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:33.326148 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:33.336134 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:33.355373 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:33.369948 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:33.404317 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:33.408049 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:33.410574 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:33.417448 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:33.417866 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:33.430683 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 08:04:33.431927 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:33.433970 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:33.514707 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:04:33.526218 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:33.527163 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:33.534201 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:33.538980 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:33.543067 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:33.546139 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:04:33.547418 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:33.559928 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:33.609747 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:33.614645 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:33.623200 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:33.623488 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:33.636528 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:33.655524 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240501 08:04:33.655828 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:33.656409 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:33.683072 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:04:33.706465 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:33.708101 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:33.718603 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:33.722473 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:33.758448 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:33.762030 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:33.771986 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:33.787863 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:33.787979 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:33.788743 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:33.789256 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:33.790575 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:33.834244 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:33.855103 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240501 08:04:33.882975 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:33.891893 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:33.896692 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:33.899068 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:33.904563 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:33.905277 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:04:33.908473 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:33.908852 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:33.982136 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:33.986181 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:33.986599 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:33.995640 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:33.999069 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:34.028295 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:34.040565 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:34.045650 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:34.055346 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:04:34.056671 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:34.060467 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:34.060732 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:34.064010 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:34.087742 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:34.088901 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:34.210150 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:34.217043 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:34.218565 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:34.225925 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:34.232007 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:34.235111 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:34.243113 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:34.253909 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:04:34.257782 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:34.263214 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:34.263294 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:34.268436 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:34.278309 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:34.281008 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:34.288143 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:34.295222 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:34.295490 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:34.305883 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:34.322909 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:34.329460 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:34.386669 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:34.433328 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:34.433943 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:34.445638 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 08:04:34.449748 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:34.453246 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:34.462333 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:34.462412 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:34.464177 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:34.469614 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:34.484503 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:34.491304 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:04:34.493402 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:34.497246 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:34.506992 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:34.544716 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:04:34.633981 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:34.638185 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:34.639384 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:34.641836 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:34.646412 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:04:34.652717 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:34.656711 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:34.671831 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:34.687511 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:34.689175 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:34.697737 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:34.703652 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:34.714497 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:04:34.726094 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:34.726636 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:34.750716 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:34.759399 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:34.770401 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:34.776031 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:34.779093 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:34.783356 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:34.825662 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:34.842634 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:34.881837 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:34.883270 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:34.884589 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:34.885939 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:34.899032 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:34.909853 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:34.929666 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240501 08:04:35.010712 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:35.011840 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:35.016327 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:35.017678 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:35.046219 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:35.048326 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:35.074596 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:35.074875 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:35.080147 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:35.086333 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:35.089216 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:35.103392 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:35.111861 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:35.129312 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:35.165699 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:35.167341 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:04:35.172797 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:35.173895 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:35.174281 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:35.180014 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:35.183144 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:35.210305 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:04:35.210659 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:35.211068 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:35.246623 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:04:35.293684 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:35.294546 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:35.294621 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:35.296029 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:35.296571 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:35.308960 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:35.363554 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:35.411958 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:35.417002 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:35.423077 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:04:35.426934 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:35.444695 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:35.444983 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:35.448415 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:35.451565 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:35.490244 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:35.493017 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:35.496310 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:35.496338 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:35.499457 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:35.524377 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:35.553776 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:35.562263 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:35.562891 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:35.563987 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:35.570041 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:04:35.570889 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:35.570981 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:35.653707 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:35.654210 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:35.656991 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:35.665643 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:35.671049 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:35.672420 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:35.679318 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:35.718736 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:35.729342 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:04:35.730139 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:35.736110 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:35.738528 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:35.739441 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:35.739877 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:35.744802 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:35.745985 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:35.779788 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:04:35.861889 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:35.864871 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:35.877116 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:35.892011 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:35.905963 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:35.922309 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:35.925094 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:35.933701 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:35.935549 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:35.937829 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:35.947991 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:35.963169 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:35.964761 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:35.979533 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:04:35.986132 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:35.987311 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:36.004076 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:36.006564 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:36.007187 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:36.027626 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 08:04:36.031905 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:04:36.036021 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:36.095821 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:36.099913 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:36.099915 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:36.100564 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:36.103202 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:36.106976 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:36.110991 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:36.140728 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:04:36.175446 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:36.178215 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:36.180459 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:36.184458 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:36.199900 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:36.208575 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:36.268227 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:36.270524 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:36.286571 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:36.286890 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:36.297021 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:36.301751 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:36.326819 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:36.335809 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:36.363510 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:36.366055 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:36.370368 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:36.382264 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:36.388875 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:36.389840 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:36.424614 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:36.426736 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:36.430718 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:36.433468 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:36.445703 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:36.446197 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:36.464444 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:04:36.536607 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:36.537820 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:36.540809 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:36.541728 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:36.544485 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:36.563388 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:36.630983 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:36.635762 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:04:36.661334 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:36.663847 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:36.666291 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:36.666791 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:36.670614 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:36.671389 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:36.817893 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:36.823006 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:36.824808 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:36.852720 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:36.853005 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:36.856173 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:36.862516 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:36.868073 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:36.869808 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:36.871186 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:36.879375 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:36.880293 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:36.881664 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:36.891670 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:04:36.891953 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:36.899535 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:04:36.909972 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:36.912954 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:04:36.922546 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:36.927760 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:36.936784 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:36.942950 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:36.958871 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:36.959370 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:37.016798 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:37.037941 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 08:04:37.047546 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:37.049868 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:37.050485 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:37.051190 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:37.054528 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:37.059947 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:37.123665 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:04:37.169404 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:37.173730 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:37.177546 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:37.178788 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:37.186036 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:37.191581 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:37.268899 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:37.312652 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:37.320746 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:37.320977 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:37.324489 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:37.333900 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:04:37.342912 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:37.351575 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:37.352478 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:37.353045 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:37.367036 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:37.379634 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:37.387684 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:04:37.391054 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:37.406296 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:37.453333 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:37.456837 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:37.460783 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:37.460798 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:37.464398 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:37.478698 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:37.528637 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:37.553886 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:37.565666 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:37.573421 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:37.574249 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:37.588232 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:37.606050 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:37.606360 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:37.607306 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:37.632190 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:37.639436 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:37.643563 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:37.643611 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:37.653393 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:37.663834 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:04:37.665179 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:37.684566 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:37.777505 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:37.783406 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:37.784137 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:37.787119 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:37.794018 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:04:37.801637 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:37.815073 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:37.815371 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:37.835314 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:37.837540 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:37.849133 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:37.849915 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:37.854151 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:37.854612 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:37.871510 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:37.873171 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:37.878801 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:37.894853 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:37.908535 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:37.914865 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:37.942469 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:37.961638 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:37.975312 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:04:38.018121 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:38.019874 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:38.023432 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:38.026657 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:38.028435 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:38.044160 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:38.052165 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:04:38.089871 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:38.090323 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:38.092362 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:38.093140 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:38.093266 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:38.100507 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:38.102618 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:38.110281 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:38.178251 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:04:38.204700 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:38.211448 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:38.216456 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:38.218250 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:38.219470 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:38.223512 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:38.243191 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:38.264500 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:04:38.270159 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:38.270370 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:38.273031 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:38.274863 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:38.275262 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:38.275830 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:38.284343 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:38.384982 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:38.386854 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:38.388980 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:38.390444 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:38.403817 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:38.419016 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:38.457063 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:38.457828 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:38.465747 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:38.467337 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:38.474030 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:38.479245 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:38.507360 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:04:38.508033 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 08:04:38.512071 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:38.559739 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:38.595635 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:38.612665 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:38.615097 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:38.623571 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:38.623663 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:38.626816 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:38.627455 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:38.627724 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:38.636806 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:38.655301 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:38.658110 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:38.660797 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:38.670596 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:38.670621 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:38.697103 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:38.706369 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 08:04:38.708070 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:38.715406 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:38.722200 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:38.722836 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:38.724376 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:38.772846 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:04:38.833125 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:38.833386 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:38.839622 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:38.846335 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:38.846343 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:38.863062 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:38.870070 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:38.877084 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:04:38.893412 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:38.894471 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:38.900837 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:38.907680 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:38.919965 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240501 08:04:38.923087 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:38.923846 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:39.023778 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:39.032651 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:39.033699 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:39.033732 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:39.048439 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:39.054617 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:39.088912 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:39.107784 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:39.109215 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:39.113597 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:39.117280 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:39.119501 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:39.119694 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:39.136207 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:04:39.149981 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:39.172410 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:04:39.237536 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:39.240178 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:39.247439 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:39.249655 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:39.256985 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:39.257221 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:39.262257 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:39.278337 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:39.280290 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:39.281327 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:39.283419 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:39.284094 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:39.296686 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:39.306174 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:39.351910 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:39.365242 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:39.412062 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:39.417405 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:39.438223 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:39.444847 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:39.459960 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:39.464215 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:39.475564 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:39.512532 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:39.513201 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:39.513341 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:39.522393 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:39.523569 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 08:04:39.529152 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:39.533212 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:39.536319 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:39.536602 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:39.542809 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:39.550529 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:39.552772 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:39.555095 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:39.583418 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:39.646701 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:04:39.680418 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:39.688828 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:39.697721 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:39.697888 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:39.712284 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:39.729183 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:39.733026 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:39.735394 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:39.749799 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:39.749893 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:39.752310 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:39.752307 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:39.855180 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:39.867534 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:39.895085 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:39.897723 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:39.902092 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:39.902289 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:39.920097 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:39.921217 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:39.928958 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:39.930085 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:39.930882 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:39.935608 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:39.954478 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:39.963414 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:39.969034 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:40.019845 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:40.021452 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:40.056067 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:04:40.072089 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:40.074525 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:40.084407 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:40.093885 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:40.098743 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:40.103076 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:04:40.109668 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:40.112727 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:40.121409 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:40.124379 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:40.128096 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:40.133366 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:04:40.133852 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:40.150009 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:40.151124 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:40.239815 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:40.246644 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:04:40.246999 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:40.256916 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:40.260655 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:40.261353 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:40.274379 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:40.284080 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:40.310365 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:40.318572 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:40.325433 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:40.325636 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:40.326108 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:40.327579 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:40.343955 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:40.388170 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:40.401023 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:40.405829 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:40.408759 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:40.414100 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:40.418474 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:40.425277 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:40.429945 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:40.493690 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:40.494910 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:40.495553 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:40.498098 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:40.500629 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:40.504485 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:40.562727 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:04:40.584316 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:40.587581 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:40.588124 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:40.594921 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:40.602528 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:40.614841 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:40.663672 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:40.669055 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:40.672487 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:40.673121 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:40.676683 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:40.686959 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:40.696942 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:40.743695 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:40.757400 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:40.761989 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:04:40.812361 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:40.814646 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:40.823617 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:40.824075 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:40.835418 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:40.838593 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:40.854734 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:04:40.855422 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:40.884299 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:40.895733 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:40.899097 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:40.900413 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:40.907832 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:40.908726 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:04:40.917371 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:40.918004 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:40.922230 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:40.947046 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:40.956739 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:40.967221 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:40.977862 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 08:04:40.979635 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:40.981274 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:41.030923 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:41.077052 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:41.077869 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:41.081465 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:41.081614 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:41.082052 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:04:41.082294 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:41.096216 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:41.109392 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:41.174674 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:41.221678 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:41.224053 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:41.230067 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:41.231858 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:41.237361 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:41.239197 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:41.260669 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:41.263160 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:41.265893 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:41.286662 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:41.286751 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:41.288547 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:41.370879 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:41.391019 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:41.412212 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:41.425598 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:41.427115 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:41.429806 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:41.434808 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:41.442665 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:41.445461 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:41.449985 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:41.454243 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:41.462378 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:41.462867 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:41.470613 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:41.521911 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:41.595757 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:41.598467 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:41.598692 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:41.599520 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:41.599779 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:41.608525 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:41.625289 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:41.644235 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:41.647306 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:41.651345 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:41.655542 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:41.655977 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:41.665331 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:41.730932 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:04:41.814407 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:41.817965 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:41.818755 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:41.830214 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:41.830670 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:41.848489 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:41.865420 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:41.866357 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:41.874081 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:41.879069 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:41.882699 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:41.887392 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:41.895819 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:41.922505 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:41.930637 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:04:41.930661 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:41.931655 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:41.954334 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:04:41.963455 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:41.975837 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:41.977255 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:42.034564 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:42.042740 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:42.046175 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:42.046890 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:42.055712 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:42.067598 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:42.071839 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:42.083587 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:42.131701 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:42.131801 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:04:42.143010 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:42.145956 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:42.150035 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:42.151705 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:42.163661 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:42.221741 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:42.221997 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:42.227344 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:42.231545 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:42.233119 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:42.243778 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:42.284339 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:42.341053 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 08:04:42.341184 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:42.345598 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:42.353327 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:42.361379 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:42.363903 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:42.364442 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:42.366812 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:42.371081 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:42.421823 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:42.423637 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:42.432757 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:42.439805 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:42.459714 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:42.478624 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:42.480441 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:42.482846 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:42.488304 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:42.491350 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:42.492789 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:42.494076 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:42.538316 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:42.549685 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:42.572758 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:42.594523 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:42.612133 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:42.620343 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:42.621784 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:42.622097 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:42.622736 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:42.633881 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:42.638475 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:42.651875 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:42.663594 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:42.677222 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:42.677711 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:42.682879 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:42.695721 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:42.702085 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:42.708495 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:04:42.790489 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:42.798538 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:42.800000 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:42.800801 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:42.800971 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:42.801615 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:04:42.824419 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:42.852486 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:42.853988 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:42.860896 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:42.863723 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:42.867734 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:42.875362 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:42.926891 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:42.996724 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:43.000080 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:43.012684 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:43.016038 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:43.016544 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:43.016779 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:43.018368 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:43.025209 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:43.029497 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:43.032356 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:43.055797 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:43.058104 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:43.059017 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:43.063558 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:43.160257 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:43.162101 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:43.171027 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:43.174468 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:43.181025 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:43.196807 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:43.203709 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:04:43.203704 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:43.236371 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:43.236366 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:43.236668 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:43.238330 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:43.240947 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:43.261586 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:43.287194 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:43.311682 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:43.340708 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 08:04:43.344576 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:43.348832 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:43.357012 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:43.361511 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:43.364423 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:43.369685 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:43.372877 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:43.376183 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:43.387723 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:04:43.438920 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:43.439872 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:43.442902 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:43.448510 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:43.448607 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:04:43.453349 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:43.454514 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:43.467401 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:43.507694 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:43.531668 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:43.531836 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:04:43.536499 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:43.543529 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:43.544051 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:43.548626 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:43.555227 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:43.564415 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:43.606062 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:43.620446 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:43.623528 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:43.627278 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:43.628841 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:43.637161 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:43.643131 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:43.654958 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:43.693207 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:43.693688 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:43.697008 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:43.698334 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:43.698472 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:43.730912 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:04:43.735059 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 08:04:43.788475 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:43.795279 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:43.799515 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:43.806633 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:04:43.816764 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:43.830600 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:43.848788 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:43.853127 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:43.863906 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:43.865684 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:43.870663 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:43.872853 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:43.882285 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:43.896414 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:43.942477 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:43.986325 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:44.018049 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:44.019104 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:44.023653 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:44.026839 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 84 similar messages]
I20240501 08:04:44.037549 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:44.040282 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:44.041554 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:44.044380 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:44.045552 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:44.059708 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:44.080147 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:44.094473 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:44.095463 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:44.100529 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:44.105023 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:44.119529 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:44.124835 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:44.128314 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:44.138211 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:44.238054 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:44.256907 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:44.257611 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:44.267562 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:44.273092 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:44.273252 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:44.278798 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:44.285796 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:44.285830 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:44.306490 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:44.306708 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:44.332485 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:44.336122 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:04:44.345913 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:44.349016 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:44.355595 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:44.424463 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:44.437992 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:44.471385 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:44.472975 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:44.478811 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:44.481559 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:44.486100 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:44.486397 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:44.486665 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:44.487092 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:44.520489 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:44.521706 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:44.527105 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:44.529435 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:44.533361 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:44.533695 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:44.543213 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:44.548322 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:44.560747 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:44.583355 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:44.673463 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:44.675205 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:44.676669 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:44.680316 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:44.680586 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:44.690280 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:44.691419 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:44.809353 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:44.818985 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 08:04:44.851884 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:44.874110 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:44.875082 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:04:44.883025 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:44.883175 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:44.889776 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:44.890081 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:44.890893 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:44.910552 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:44.937420 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:44.939044 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:44.951730 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:44.953166 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:44.953164 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:44.967931 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:45.008450 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:45.063962 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:04:45.064906 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:45.065964 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:04:45.067019 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:45.069645 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:45.075143 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:45.089610 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:45.090932 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:45.120702 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:45.121546 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:45.125706 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:45.129396 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:45.130375 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:45.145080 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:45.145673 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:45.164817 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:45.195505 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:04:45.208266 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:45.258090 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:45.265306 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:45.266676 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:45.273838 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:45.307972 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:45.308207 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:45.309865 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:45.320070 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:04:45.320633 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:45.321133 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:45.332875 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:45.345963 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:45.355528 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:45.359373 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:45.360903 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:04:45.361258 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:45.361851 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:45.363246 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:45.365290 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:45.374277 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:45.393492 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:45.399574 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:45.494235 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:45.503688 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:45.505571 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:45.514724 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:45.515614 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:45.526891 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:45.532351 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:45.539099 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:45.544919 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:45.545317 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:45.547813 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:45.559350 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:45.581981 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:45.644044 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:45.692210 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:45.695264 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:45.698030 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:45.706679 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:45.711975 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:45.714010 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:45.723403 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:04:45.744887 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:45.762558 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:45.777324 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:45.791221 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:04:45.791790 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:45.796955 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:45.800071 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:45.805848 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:45.807248 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:45.807507 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:04:45.811345 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:45.830183 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:45.836076 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:04:45.836427 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:45.838850 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:45.843108 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 08:04:45.844759 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:04:45.846232 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:45.856787 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:45.916419 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 08:04:45.955451 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:45.959071 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:45.960911 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:45.966375 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:45.975021 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:45.982324 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:45.984915 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:45.995648 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:46.002331 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:46.002732 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:46.020761 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:46.033058 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:46.033968 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:04:46.034644 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:46.130152 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:46.137398 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:46.138774 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:46.139762 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:46.158807 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:46.187181 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:46.213711 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:46.219228 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:46.219607 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:46.221365 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:46.222131 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:46.244421 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:46.244491 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:46.265367 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:46.266211 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:46.277266 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:46.283618 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:46.301002 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:46.304391 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:46.322494 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:04:46.357848 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:46.361763 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:46.364398 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:46.365062 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:46.377489 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:46.391624 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:46.445991 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:46.449061 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:46.450214 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:46.451196 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:46.452769 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:46.462365 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:46.464129 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:46.502060 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:46.562374 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:46.562922 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:46.569073 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:46.572845 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:46.579736 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:46.582459 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:46.592423 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:46.663739 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:46.685796 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:46.687240 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:46.693003 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:46.693720 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:46.696228 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:46.698328 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:46.797693 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:46.800773 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:46.805655 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:46.807020 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:46.818557 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:46.822058 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:46.852092 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:04:46.857869 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:46.865993 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:46.870563 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:46.872514 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:46.877895 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:46.878171 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:46.878551 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:46.914124 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:46.976696 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:46.990460 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:47.000130 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:47.000429 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:47.002203 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:47.015991 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:47.039611 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:47.062407 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:47.063542 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:47.064728 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:47.069195 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:47.074699 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:47.095036 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:47.129508 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:47.132998 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:47.137375 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:47.141948 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:47.144639 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:47.149286 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:47.160645 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:47.244027 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:47.246764 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:47.247107 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:47.247222 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:47.269747 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:47.282761 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:47.313210 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:04:47.316282 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:47.321029 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:47.321488 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 08:04:47.326110 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:47.332118 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:47.333156 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:47.335747 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:47.347321 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:47.351065 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:47.355288 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:04:47.411147 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:47.435313 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:47.443939 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:47.454216 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:47.456262 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:47.467711 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:47.476159 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:47.484515 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:47.528642 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:47.531966 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:47.538623 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:47.540150 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:47.542232 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:47.547922 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:47.555507 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:47.571933 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:47.574080 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:47.581002 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:47.588821 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:47.617072 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:47.634079 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:47.668601 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:04:47.712277 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:47.719420 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:47.720486 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:47.737159 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:47.747509 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:47.748903 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:47.759276 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:47.772543 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:47.773953 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:47.775591 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:04:47.781507 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:47.797720 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:47.809490 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:47.812248 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:47.821652 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:47.861757 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:47.863408 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:04:47.944036 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:47.955984 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:47.962574 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:47.975549 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:47.980101 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:47.981491 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:47.997393 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:47.998349 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:48.005707 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:48.006786 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:48.007509 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:48.016090 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:48.018822 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:48.023550 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:04:48.025481 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:48.028311 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:48.031548 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:48.041568 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:48.042579 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:48.060426 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:48.159826 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:04:48.163807 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:48.166582 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:48.167829 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:48.176941 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:48.180047 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:48.192461 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:48.200966 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:04:48.202313 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:04:48.211930 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:48.286079 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:48.290062 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:48.291026 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:48.293699 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:48.293702 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:48.304184 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:48.318630 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:04:48.319478 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:48.370507 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:48.394810 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:48.399420 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:48.406348 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:48.407930 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:48.421119 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:48.424746 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:48.438380 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:48.447692 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:48.448107 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:48.448463 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:48.457185 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:48.461932 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:48.462007 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 08:04:48.476926 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:04:48.480793 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:04:48.492288 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:48.574960 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:04:48.575660 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:48.576259 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:48.578034 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:48.581780 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:48.608386 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:48.611929 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:48.666980 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:48.669515 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:48.670078 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:48.673390 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:48.675459 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:48.677412 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:48.684170 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:48.757021 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:48.758843 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:48.762907 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:48.770329 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:48.775223 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:48.783414 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:48.800557 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 76 similar messages]
I20240501 08:04:48.802822 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:48.861018 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:48.864681 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:48.875536 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:48.883962 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:48.886194 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:48.886963 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:48.896037 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:48.930084 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:48.931807 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:48.947610 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:48.947788 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:48.950068 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:48.950309 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:48.961104 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:48.978353 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:49.018754 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:49.019227 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:49.026165 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:49.027431 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:49.034353 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:49.047144 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:04:49.047730 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:49.110122 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:49.113268 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:49.137817 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:49.144304 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:49.144757 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:49.150476 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:49.156160 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240501 08:04:49.218533 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:49.220278 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:49.221043 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:49.221683 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:49.237820 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:49.241923 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:49.246150 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:49.263528 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:49.310561 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:49.320327 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:49.326818 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:49.328624 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:49.329167 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:49.345315 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:49.352286 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:49.429227 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:49.429466 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:49.439074 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:49.443037 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:49.450027 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:49.451146 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:49.453225 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:49.456966 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:49.459609 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:49.461273 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:49.465936 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:49.479887 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:49.480471 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:49.493847 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:49.565527 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:49.567886 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:49.569072 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:49.570966 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:49.579349 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:49.589529 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:49.590821 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:49.637570 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:49.661206 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:49.663995 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:49.666332 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:49.668026 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:49.681758 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:49.683997 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:49.684985 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:49.749050 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:49.754765 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:49.755285 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:49.756381 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:49.763731 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:49.765604 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:49.774802 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:49.779260 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:49.863828 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:49.865486 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:49.876555 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:49.885912 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:49.886561 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 08:04:49.887300 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:49.887818 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:04:49.911897 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:49.921073 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:49.936229 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:49.936326 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:49.936841 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:49.947824 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:49.948150 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:49.969408 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:50.047369 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:50.047634 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:50.048642 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:50.049804 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:50.050732 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:50.052685 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:50.064083 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:50.119598 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:50.120880 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:50.131327 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:50.132325 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:50.133976 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:50.152932 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:50.187944 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 08:04:50.196575 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:50.208169 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:50.209913 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:50.221704 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:50.224083 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:50.224781 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:50.234777 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:50.249604 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:50.291534 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:50.299281 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:50.299526 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:50.301131 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:50.302832 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:50.321228 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:04:50.328053 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:50.368427 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:50.383842 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:50.388947 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:50.388986 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:50.392522 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:50.410888 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:50.426015 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:50.458262 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:04:50.461010 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:50.478072 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:50.494992 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:50.496557 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:50.498571 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:50.510303 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:50.519570 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:50.530274 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:50.577044 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:50.577607 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:50.577944 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:50.590010 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:50.599882 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:50.600306 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:50.601079 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:50.608845 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:50.639233 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:04:50.651605 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:04:50.681389 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:50.696113 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:50.698056 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:50.712062 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:50.712890 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:50.716095 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:50.718463 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:50.729362 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:50.730580 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:50.739544 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:50.757560 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:50.758515 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:50.763952 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:50.777861 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:50.779381 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:50.860386 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:50.861497 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:50.864804 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:50.868616 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:50.874831 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:50.883661 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:50.896461 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:50.897620 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:50.902242 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:50.923401 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:50.925765 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:50.932858 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:50.932859 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:50.935263 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:50.936369 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:50.960487 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:50.978866 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 08:04:50.979704 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:51.053615 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:51.054775 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:51.059430 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:51.059816 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:51.072355 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:51.079803 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:51.089392 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:51.098400 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:51.103497 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:51.108801 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:51.133801 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:51.136317 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:51.235867 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:51.241667 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:51.242657 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:51.243927 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:51.248144 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:51.264703 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:51.276577 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:51.283398 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:51.329835 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:51.333289 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:51.333389 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:51.335150 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:51.335736 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:51.352372 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:51.353951 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:51.394294 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:51.396189 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:51.399112 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:51.400617 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:51.402985 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:51.403981 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:51.420017 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:51.485518 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:51.514762 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:51.514922 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:51.515324 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:51.518385 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:51.519608 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:51.521461 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:51.532095 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:51.537930 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:51.635344 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:51.635560 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:51.636209 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:51.651149 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:51.656282 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:51.656283 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:51.710578 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:51.714879 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:51.722132 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:51.730620 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:51.732079 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:51.733021 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:51.736774 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:51.755393 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:51.763986 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:51.778849 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:51.780627 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:51.784413 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:51.797230 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:51.805231 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:51.898756 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:51.898977 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:51.913571 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:51.916838 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:51.925756 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:51.936108 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:51.942462 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:51.962762 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:51.962814 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:51.974781 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:51.988754 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:51.990013 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:51.998492 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:52.015095 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:04:52.050482 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:52.081380 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:52.083735 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:52.087800 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:52.090098 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:52.105171 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 08:04:52.108554 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:52.108696 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:52.110113 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:52.160559 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:52.163179 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:52.166635 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:52.167024 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:52.169405 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:52.187062 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:52.273417 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:52.282902 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:52.291081 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:52.309125 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:52.315483 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:52.316576 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:52.318459 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:52.323917 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:52.330026 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:52.337074 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:52.358958 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:52.361371 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:52.369558 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:52.374179 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:52.392740 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:04:52.403609 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:52.454473 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:52.456516 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:52.458171 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:52.461804 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:52.468972 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:52.479091 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:52.491324 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:52.518625 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:52.518651 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:52.530300 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:52.530364 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:52.530395 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:52.535367 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:52.553083 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:04:52.563118 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:52.598737 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:52.603056 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:52.608732 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:52.609166 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:52.610638 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:52.617267 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:52.639402 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:52.713171 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:52.715200 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:52.719628 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:52.727491 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:52.740772 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:52.744933 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:52.748031 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:52.775969 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:52.781203 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:52.784799 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:52.789310 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:52.790261 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:52.798599 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:52.801754 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:52.880450 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:52.920949 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:52.926962 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:52.930233 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:52.933529 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:52.940145 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:52.945127 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:52.949853 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:52.962399 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:52.962905 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:52.963171 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:52.965405 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:52.982470 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:52.999280 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:53.038594 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:53.052330 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:53.058086 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:53.059541 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:53.060542 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:53.064035 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:53.070153 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:53.071990 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:53.136886 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:53.139325 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:53.140753 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:53.142549 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:53.150965 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:53.151767 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:53.166211 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:53.169754 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:53.198711 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:04:53.222506 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:04:53.234488 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:04:53.259968 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:53.260423 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:53.262068 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:53.262173 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:53.270987 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:53.271268 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:53.280859 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:53.352023 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:53.357969 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:53.358202 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:04:53.366329 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:53.370323 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:53.378122 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:53.379302 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:53.384966 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:53.390841 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:53.433924 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 08:04:53.460803 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:53.461570 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:53.461580 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:53.466338 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:53.468163 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:53.481627 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:53.484745 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:53.502899 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:53.504698 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:53.510097 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:53.527173 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:53.528422 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:53.534065 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:04:53.538725 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:04:53.592746 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:53.629284 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:53.634687 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:53.640100 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:53.645937 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:53.647118 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:53.660531 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:53.671759 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:53.683053 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:53.683310 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:53.684473 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:53.694669 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:53.707538 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:53.734227 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:53.792721 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:53.800011 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:53.800953 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:53.808625 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:53.813311 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 83 similar messages]
I20240501 08:04:53.839713 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:53.846813 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:53.851529 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:53.865273 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:53.869172 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:53.870122 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:53.870482 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:53.880079 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:53.942703 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:04:53.974850 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:53.988917 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:53.990872 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:53.991204 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:53.994364 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:53.994549 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:53.994817 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:54.023032 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:54.068413 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:54.071570 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:54.071584 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:54.074769 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:54.077693 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:54.082656 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:54.087644 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:54.140144 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:04:54.175067 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:54.176295 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:54.183681 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:54.185428 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:54.207804 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:54.209265 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 08:04:54.210821 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:04:54.215343 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:54.215889 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:54.256860 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:54.263788 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:54.266078 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:54.274206 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:54.278575 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:54.287120 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:54.291538 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:54.295938 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:54.297557 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:54.307173 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:54.332883 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 08:04:54.333464 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:54.342881 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:54.398116 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:54.403487 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:54.404937 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:54.420161 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:54.423763 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:54.435346 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:54.440101 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:54.486011 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:54.486878 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:54.487465 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:54.488283 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:54.494164 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:54.509456 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:54.523029 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:54.587146 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:54.587944 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:54.601455 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:54.603997 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:54.606817 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:54.616412 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:54.617667 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:54.671548 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:54.677863 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:54.678767 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:54.681399 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:54.690609 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:54.693177 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:54.695820 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:54.696092 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:54.721936 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:54.738587 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:54.740077 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:54.740563 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:54.742972 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:54.749872 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:54.771538 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:54.822455 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:04:54.846472 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:04:54.852226 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:54.859301 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:54.860373 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:54.860769 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:54.862812 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:54.871461 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:54.885355 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:54.971503 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:54.972484 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:54.975924 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:54.981106 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:54.981489 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:54.982546 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:55.012651 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:55.044620 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:55.048933 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:55.053694 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:55.059196 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:55.061256 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:55.077098 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:55.123950 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:55.152828 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:55.160717 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:55.172165 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:55.175835 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:55.178568 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:55.178570 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:55.188937 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:55.195201 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:55.198216 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:55.205201 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:55.208760 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:55.213245 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:55.213477 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:55.223030 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:55.242307 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:55.260926 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:04:55.264490 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:55.299670 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:55.299939 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:55.304162 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:55.317332 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:55.318583 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:55.320058 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:55.337795 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:55.403208 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:55.407253 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:55.408591 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:55.411031 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:55.414688 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:55.420503 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:55.462383 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:04:55.478685 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 08:04:55.488642 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:55.488787 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:55.495798 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:55.500892 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:55.503762 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:55.514732 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:55.527149 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:55.528749 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:55.559299 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:04:55.576498 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:04:55.585323 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:55.588284 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:55.609460 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:55.611063 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:55.616617 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:55.621847 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:55.624946 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:55.625221 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:55.631143 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:55.634778 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:55.643545 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:55.653193 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:55.664328 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:55.769829 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:55.773626 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:55.773608 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:55.773959 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:55.774612 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:55.782164 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:55.793663 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:55.872139 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:55.880694 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:55.883064 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:55.883790 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:55.883908 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:55.893846 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:55.896090 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:04:55.897081 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:55.899037 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:55.950719 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:55.967093 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:55.968958 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:55.972083 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:55.972738 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:55.976740 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:55.985384 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:56.007581 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:56.077992 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:56.081648 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:56.092540 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:56.095105 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:56.098889 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:56.104326 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:56.104570 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:56.111109 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:56.115092 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:56.119922 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:56.124611 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:56.133594 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:56.142817 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:56.201846 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240501 08:04:56.205060 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:56.205948 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:56.210383 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:56.222286 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:56.224354 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 08:04:56.227463 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:56.235065 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:56.311889 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:56.313987 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:56.315341 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:56.322460 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:56.323029 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:04:56.329020 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:56.335881 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:56.344959 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:56.371541 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:56.433028 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:56.435448 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:56.437093 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:56.437288 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:56.447796 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:56.451143 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:56.490327 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:04:56.513249 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:56.516162 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:56.521952 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:56.524013 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:56.527590 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:56.545346 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:04:56.551196 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:56.555227 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:56.611555 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:56.611975 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:56.612738 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:56.618021 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:56.624460 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:56.638582 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:56.684523 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:56.684523 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:56.689746 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:56.699045 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:56.705662 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:56.705660 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:56.719548 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:56.786545 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:56.791468 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:56.800324 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:56.801210 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:56.806047 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:56.818847 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:56.838657 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 08:04:56.871788 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:56.886082 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:56.887337 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:56.890195 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:56.895987 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:56.926381 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:56.944763 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:56.954293 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:56.959352 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:56.985141 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:56.988667 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:56.995561 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:56.996735 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:57.056123 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:57.061703 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:57.063271 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:57.071055 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:57.071244 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:57.071296 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:57.071310 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:57.104571 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:04:57.129099 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:57.133107 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:57.137068 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:57.142793 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:57.149526 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:57.164306 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:04:57.171686 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:57.203514 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:04:57.218387 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:57.218848 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:57.223587 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:57.227609 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:57.242611 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:57.247411 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:57.252245 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:04:57.260751 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:04:57.325649 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:57.332142 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:57.332132 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:57.334048 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:57.346385 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:57.346720 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:57.377631 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:57.429498 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:57.429491 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:57.430670 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:57.430692 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:57.432173 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:57.440583 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:57.445351 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:57.446568 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:57.451099 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:57.457326 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:57.473865 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:57.493362 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:57.494236 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:57.542393 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:04:57.566648 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:57.578174 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:57.579274 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:57.579525 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:57.580758 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:57.582629 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:57.583287 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:57.645563 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:04:57.689882 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:57.692934 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:04:57.696293 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:57.701139 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:04:57.705710 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:57.709400 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:57.714565 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:57.742071 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:57.758610 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:57.763087 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:57.768786 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:57.769840 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:57.784747 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:57.790673 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:57.809648 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:57.835016 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 08:04:57.843312 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:04:57.867116 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:57.867698 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:57.870406 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:57.874912 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:57.880095 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:57.880612 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:57.895965 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:04:57.908497 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:04:57.909862 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:57.937582 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:57.943012 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:57.944674 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:57.947577 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:57.967706 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:57.969482 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:58.051640 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:04:58.055732 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:58.059691 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:58.060037 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:58.060405 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:58.062247 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:58.066004 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:58.066449 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:58.172506 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:58.178838 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:58.187636 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:58.189841 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:58.190970 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:58.192525 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:58.195950 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:58.219492 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:58.226621 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:58.232439 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:58.237764 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:58.244122 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:58.248755 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:58.260059 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:04:58.268172 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:58.270565 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:58.273942 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:58.320267 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:58.321205 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:58.323009 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:58.326730 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:58.329433 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:58.335844 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:58.411615 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:58.412624 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:58.414673 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:58.415225 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:58.417656 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:58.418601 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:58.422899 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 08:04:58.424058 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:58.441495 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:58.447494 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:04:58.484705 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:58.484706 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:58.486588 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:58.492391 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:58.492686 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:58.499884 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:58.515285 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:58.593978 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:58.603086 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:58.605144 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:58.605348 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:58.613024 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:58.613461 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:58.666239 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:58.668166 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:58.673219 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:58.683777 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:58.690495 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:58.690852 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:58.749890 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:04:58.757457 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:58.765645 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:58.772240 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:58.773612 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:58.778599 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:58.778879 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:58.807317 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:04:58.818934 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 08:04:58.864066 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:58.866288 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:58.868558 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:58.875900 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:58.878541 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:58.887086 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:58.895218 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:04:58.932247 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:04:58.937229 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:58.937523 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:58.942453 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:58.952865 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:58.953269 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:58.967175 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:04:58.975924 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:58.987284 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:59.031072 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:59.031955 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:59.047627 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:59.057034 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:59.057564 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:59.063508 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:59.139968 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:59.149276 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:59.156312 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:59.156613 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:59.158461 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:59.159925 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:59.163570 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 08:04:59.196240 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:59.196944 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:04:59.202630 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:59.209254 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:59.215190 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:59.223731 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:59.231663 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:59.296175 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:59.298219 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:59.299772 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:59.301271 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:59.314960 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:59.322996 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:59.340797 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:04:59.365562 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 08:04:59.396050 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:59.415084 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:59.415408 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:59.415519 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:59.416726 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:59.421149 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:59.434365 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:04:59.462503 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:04:59.477908 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:59.483049 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:59.483089 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:59.487318 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:59.489836 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:59.510607 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:59.550565 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:59.588225 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:59.604511 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:59.613608 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:59.614157 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:59.616608 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:59.616863 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:59.620138 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:59.622198 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:59.627502 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:59.640178 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:04:59.643517 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:04:59.643942 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:04:59.671819 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:59.674296 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:04:59.736985 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:04:59.760685 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:59.771275 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:04:59.771549 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:59.773949 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:59.775010 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:59.775010 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:59.793596 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:04:59.861316 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:04:59.862579 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:04:59.865634 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:04:59.871290 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:59.873261 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:04:59.873469 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:04:59.878657 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:04:59.892814 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:04:59.896081 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:04:59.970517 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:04:59.971967 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:04:59.972570 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:04:59.976862 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:04:59.985582 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:00.011657 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:00.022886 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:05:00.031145 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:00.037379 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:00.041332 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:00.045079 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:00.057870 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:00.073184 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:00.085999 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:00.089282 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:00.093713 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:00.100168 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:00.101174 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:00.101598 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:00.115943 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:00.181586 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:00.194350 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:00.233167 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:05:00.234149 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:00.234442 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:00.237733 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:00.241869 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:05:00.242991 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:00.253651 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:00.253865 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:00.257102 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:00.257239 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:00.260026 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:00.291967 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:00.293296 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:00.304172 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:00.362219 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:05:00.413435 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:00.418797 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:00.434717 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:00.438689 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:00.438967 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:00.448808 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:00.456588 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:00.459249 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:05:00.473793 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:05:00.475771 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:00.479990 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:00.475782 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:05:00.483425 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:00.484930 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:00.487828 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:00.529500 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:05:00.554530 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:00.554733 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:00.561655 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:00.569598 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:00.570585 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:00.571974 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:00.594905 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:00.657480 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:00.663378 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:00.667503 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:00.668287 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:00.670009 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:00.675050 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:00.681058 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:00.697386 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:00.708736 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:00.711344 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:00.725374 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:00.735324 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:00.804541 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:00.810549 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:00.820492 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:00.824216 13385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:05:00.851496 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:00.857532 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:00.860371 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:00.875160 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:05:00.876121 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:00.878015 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:00.889782 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:00.890610 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:00.890758 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:00.894831 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:00.905499 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:00.907436 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:00.925472 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:05:00.929169 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:00.941478 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:00.984349 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:00.995932 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:00.995932 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:01.005560 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:01.010478 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:01.015059 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:01.066604 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:01.069401 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:01.069623 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:01.075520 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:01.076444 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:01.087584 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:01.122040 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:05:01.151518 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 08:05:01.209182 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:05:01.246623 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:01.248379 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:01.256656 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:01.258796 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:05:01.259330 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:05:01.262600 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:01.268640 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:01.385025 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:01.387284 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 08:05:01.406903 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:01.407403 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:01.407866 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:01.415521 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:01.440023 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:01.442013 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:01.446964 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:01.448249 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:01.453588 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:01.456032 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:01.462808 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:01.476869 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:01.485396 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:01.502938 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:01.507645 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:01.513003 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:01.519447 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:05:01.526247 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:01.529145 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:01.547549 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:01.652220 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:01.652585 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:01.655869 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:01.658622 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:01.660869 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:01.661926 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:05:01.672870 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:01.683822 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:01.686128 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:01.691624 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:01.692121 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:01.705984 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:05:01.711978 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:01.722074 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:01.728171 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:01.832903 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:01.847906 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:01.850600 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:01.865540 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:01.879689 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:01.887074 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:01.888321 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:05:01.899972 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 08:05:01.906709 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:01.906970 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:01.909178 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:01.929757 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:01.937602 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:01.938787 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:01.944447 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:01.946560 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:01.971632 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:01.971668 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:01.975040 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:02.006633 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:02.086874 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:02.088827 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:02.089725 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:02.098814 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:02.103013 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:02.135814 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:02.147254 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:05:02.148780 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:02.151345 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:02.157850 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:02.160107 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:02.163879 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:02.165004 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:02.214839 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:02.264225 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:02.288378 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:02.296500 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:02.299425 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:05:02.301120 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:02.304299 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:02.312601 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:02.315203 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:02.325381 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:05:02.334468 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:02.338732 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:02.347723 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:02.360321 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:05:02.372125 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:05:02.390295 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:02.393216 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:02.393525 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:02.409003 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:02.410718 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:02.411677 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:02.414786 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:02.426211 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:02.431074 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:05:02.492436 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:02.513971 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:02.533521 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:05:02.538666 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:02.541785 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:02.542172 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:02.551975 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:02.555002 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:05:02.576961 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:02.578137 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:02.580173 13428 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:02.580250 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:02.582731 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:02.582952 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:02.590790 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:02.595504 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:02.660439 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:05:02.684152 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:02.736567 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:05:02.742511 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:02.749743 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:02.754685 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:05:02.757606 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:02.758569 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:02.760964 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:02.792616 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:02.796999 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:05:02.804935 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:02.810842 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:02.812939 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:02.818593 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:02.821506 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:02.822584 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:02.839040 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:02.841786 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:02.843302 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:02.851300 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:02.877665 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:05:02.886575 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:05:02.900302 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:02.941226 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:02.965802 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:05:03.036809 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:05:03.124056 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:05:03.195669 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:03.202625 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:03.203697 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:03.205360 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:03.216634 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:03.229422 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:03.260526 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:03.261394 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:03.266233 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:03.287967 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:03.295019 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:03.298017 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:03.299520 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:03.300475 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:03.303185 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:03.307483 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:03.316263 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:03.335805 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:03.337757 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:03.430716 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:05:03.438360 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:03.441219 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:03.458942 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:03.459049 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:03.480101 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:03.484185 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:03.493791 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:03.497535 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:03.502626 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:03.503577 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:03.505710 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:03.510222 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:03.517930 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:03.524770 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:05:03.535632 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:03.603478 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 08:05:03.613853 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 08:05:03.615315 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:03.632089 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:03.633841 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:03.640321 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:05:03.644500 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:03.644521 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:03.645301 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:03.669981 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:03.705207 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:03.707197 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:03.721928 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:03.729136 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:03.733666 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:03.743255 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:03.747522 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:03.758030 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:03.761209 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:03.767429 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:03.777861 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:03.792107 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:03.837755 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 77 similar messages]
I20240501 08:05:03.876593 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:03.878335 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:03.880209 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:03.882140 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:03.889796 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:03.891459 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:05:03.921903 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:03.965124 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:03.966878 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:05:03.967586 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:03.968122 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:03.968448 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:03.968917 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:03.977746 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:03.992857 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:03.998816 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:05:04.034207 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:04.035029 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:04.054903 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:04.061944 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:04.068528 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:04.079761 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:04.081365 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:04.091382 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:04.091750 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:04.140206 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:05:04.172443 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:04.173929 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:04.177090 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:04.188901 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:04.193269 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:04.193466 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:04.194047 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:04.218654 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:04.222754 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:05:04.226686 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:05:04.229382 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:04.229990 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:04.233520 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:04.311502 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:04.316824 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:04.318758 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:05:04.319963 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:04.319979 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:04.323675 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:04.333482 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:05:04.362370 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 08:05:04.368590 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 08:05:04.395344 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:04.397045 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:04.403530 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:04.417280 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:04.427364 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:04.429905 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:04.480831 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:04.484709 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:04.486941 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:04.494612 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:04.499998 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:04.500268 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:04.518704 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:05:04.565165 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:04.588320 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:04.588407 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:04.590548 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:04.590675 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:04.599309 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:04.605335 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:04.664595 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:04.667403 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:04.670753 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:04.671317 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:04.673800 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:04.690591 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:04.732146 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:04.734973 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:04.736595 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:04.743568 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:04.749666 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:04.753486 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:04.758035 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:04.782153 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:04.893004 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:04.895603 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:04.923892 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:04.925014 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:04.930392 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:04.932184 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:04.940800 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:04.953509 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:05:04.960304 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:04.963943 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:05:05.011271 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:05:05.030529 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:05.036105 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:05.037619 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:05.043350 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:05.046350 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:05.051012 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:05.055619 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:05:05.056227 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:05.121572 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:05.122216 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:05.132912 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:05.138679 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:05.145902 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:05.147048 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:05.164698 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:05.167120 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:05.167862 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:05.171268 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:05.171545 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:05.177691 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:05.180589 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:05:05.181663 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:05.197880 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:05.270052 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:05:05.323385 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:05.326277 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:05:05.328096 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:05.332079 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:05.336467 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:05:05.337556 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:05.340083 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:05.341111 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:05.342854 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:05.352310 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:05.386512 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:05.409235 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:05.412726 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:05.425004 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:05.463379 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:05.469645 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:05.471334 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:05.472782 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:05.484236 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:05.485292 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:05.487413 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:05.493446 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:05.530166 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:05.552172 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:05.552593 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:05.553826 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:05.561431 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:05.574658 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:05.581044 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:05.625988 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:05.627149 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:05:05.633015 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:05.634690 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:05.636191 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:05.645829 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:05.650635 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:05.654825 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:05:05.744180 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:05.748415 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:05.751005 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:05.755722 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:05.765774 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:05.767457 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:05.825191 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:05.827015 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:05.828625 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:05.831614 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:05.833076 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:05.833999 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:05:05.863078 13384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:05:05.863276 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:05.950325 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:05.957767 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:05.958195 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:05.961308 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:05:05.970273 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:05:05.976747 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:05.979573 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:05.993893 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:06.004568 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:06.011241 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:05:06.012961 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:06.020188 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:05:06.027779 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:06.031486 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:06.035089 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:06.035706 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:06.057088 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:06.057834 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:06.058625 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:06.092204 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:06.093824 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:06.106974 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:06.121973 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:05:06.167086 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:06.173236 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:06.179653 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:06.185089 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:06.186564 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:06.188306 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:06.213208 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:06.254098 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:06.255409 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:06.257023 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:06.259370 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:06.273552 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:06.274262 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:06.283035 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:06.312578 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:05:06.335793 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:06.335837 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:06.340204 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:06.342916 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:06.346395 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:06.346674 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:06.405757 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 08:05:06.406373 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:05:06.418795 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:05:06.450588 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:06.452140 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:06.455013 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:06.461043 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:06.467615 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:06.501703 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:06.502347 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:06.502660 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:06.504599 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:06.515254 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:06.521546 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:06.533073 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:06.535032 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:06.535143 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:06.583361 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:05:06.593066 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 08:05:06.605938 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:06.609642 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:06.613886 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:06.617079 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:06.623261 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:06.625293 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:06.677109 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:06.714661 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:06.715059 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:06.721714 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:06.722294 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:06.728546 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:06.730037 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:06.865398 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:06.871126 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:06.871730 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:06.874682 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:06.875746 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:06.898917 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:06.918488 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:06.921425 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:06.924753 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:06.925343 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 08:05:06.926837 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:06.932582 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:06.942132 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:06.976469 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:06.986176 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:06.987221 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:06.987464 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:06.989797 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:06.990757 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:06.996325 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:07.064656 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:07.097168 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:07.100396 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:07.102334 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:07.102524 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:07.107069 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:07.113703 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:07.155113 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:07.170575 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:07.173836 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:07.178401 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:07.195896 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:07.214013 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:07.214747 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:07.216706 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:07.268868 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:07.270782 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:07.271135 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:07.275002 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:07.298100 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:07.302244 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:07.338528 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:07.346714 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:07.351119 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:07.365573 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:07.367681 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:07.369172 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:07.438103 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:07.439925 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:07.440450 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:07.441581 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:07.453848 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:07.454380 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:07.458155 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:07.493286 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:05:07.524716 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:07.525099 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:07.525666 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:07.535535 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:07.537616 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:07.539645 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:07.540616 11467 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:05:07.555461 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:05:07.584530 11467 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:07.585158 11205 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:07.615820 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:07.624707 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:07.638075 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:07.643725 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:07.651101 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:07.655997 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:07.666028 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:05:07.692011 11337 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:05:07.692312 13121 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:05:07.719410 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:07.723156 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:07.726382 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:07.727373 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:07.727914 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:07.750664 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:07.771742 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:07.779510 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:07.794636 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:05:07.809396 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:07.809656 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:07.816740 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:07.834626 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:07.843740 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:07.843957 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:07.847077 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:07.895886 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:07.908041 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:07.908528 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:07.911831 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:07.927964 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:07.936333 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:07.952682 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:07.978557 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:07.978989 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:07.981747 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:07.985823 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:07.994172 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:08.014070 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:08.015924 11205 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:08.107611 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:08.114073 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:08.121295 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:08.127770 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:08.134516 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:08.136566 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:08.162016 11206 consensus_peers.cc:485] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:05:08.204260 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:08.207756 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:08.213872 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:08.216296 11206 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:05:08.216432 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:08.228560 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:08.241465 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:08.275398 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:08.305999 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:08.308604 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:08.308993 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:08.329550 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:08.330976 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:08.332998 11205 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:05:08.349937 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:08.363809 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:08.364885 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:08.367025 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:08.384268 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:08.397452 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:08.402220 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:08.448237 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:08.494582 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:08.494784 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:08.502022 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:08.502599 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:08.502885 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:08.503696 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:08.591948 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:08.593945 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:08.596535 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:08.600636 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:08.633809 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:08.642526 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:08.650014 11468 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:05:08.685384 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:08.686559 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:08.692059 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:08.699091 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 08:05:08.705519 11336 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:05:08.710959 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:08.718616 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:08.720629 11205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 08:05:08.721379 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:08.721494 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:05:08.745918 11598 consensus_peers.cc:485] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:05:08.774271 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:08.776919 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:08.780197 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:08.787523 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:08.788491 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:08.795192 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:08.860076 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:08.880257 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:08.884984 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:08.889014 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240501 08:05:08.892686 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:08.893362 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 08:05:08.907024 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:05:08.928109 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:08.936095 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:08.952992 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:08.963510 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:08.964100 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:08.970080 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:08.980783 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:08.983597 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:08.995177 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:09.001776 11599 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 08:05:09.019315 11205 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:09.065768 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:09.069267 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:09.086243 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:09.089751 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 08:05:09.106220 11467 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 08:05:09.108470 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:09.115550 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:09.117766 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:09.120728 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:09.122833 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:09.148321 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:09.154451 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:09.154717 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:09.179250 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:09.208009 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:09.209628 11336 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:05:09.222993 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:09.223639 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:09.233740 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:09.234215 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:09.243916 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:09.259846 13384 consensus_peers.cc:485] T 53193305f868433c852755e8a8a81766 P ca177138f2c44d2eafa1646b795f7608 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 08:05:09.259904 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:05:09.316180 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:09.317960 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:09.322912 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:09.324149 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:09.331916 11205 consensus_peers.cc:462] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:09.334354 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:09.338603 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:09.400070 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:09.401569 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:09.404204 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:09.417506 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:09.420513 11599 consensus_peers.cc:462] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:09.430325 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:09.436030 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:09.462919 13385 consensus_peers.cc:485] T f2e63d24d7df4fa3b2b26695e70ebe9f P ca177138f2c44d2eafa1646b795f7608 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 08:05:09.476246 11467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 08:05:09.498260 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:09.514838 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:09.515465 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:09.517110 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:09.527084 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:09.546025 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:09.598032 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:09.601974 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:09.604205 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:09.606953 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:09.617213 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:09.636837 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:09.637102 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:09.639665 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:09.639976 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:09.647604 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:09.652086 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:05:09.654947 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:09.670863 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:09.706526 31786 raft_consensus.cc:998] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05: Attempting to remove follower 60e8857abe7c41038af46ebb5a095a90 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 60e8857abe7c41038af46ebb5a095a90 for more than 300 seconds (300.074s)
I20240501 08:05:09.706990 31786 consensus_queue.cc:237] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:05:09.708411 13448 raft_consensus.cc:1261] T c95c4f3ef08f44d1b69fb7b98426f5af P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:05:09.708839 11531 raft_consensus.cc:1261] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:05:09.709256 31786 consensus_queue.cc:1035] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:05:09.709463 32388 consensus_queue.cc:1035] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:05:09.711454 31786 raft_consensus.cc:2932] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:09.711972 13448 raft_consensus.cc:2932] T c95c4f3ef08f44d1b69fb7b98426f5af P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:09.712555 11531 raft_consensus.cc:2932] T c95c4f3ef08f44d1b69fb7b98426f5af P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:09.714898 11126 catalog_manager.cc:5484] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
W20240501 08:05:09.720717 11114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:05:09.723557 11114 catalog_manager.cc:4573] TS 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c95c4f3ef08f44d1b69fb7b98426f5af: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:09.775048 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:09.783039 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:09.783845 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:09.792495 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:09.795643 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:09.800042 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:09.822463 31786 raft_consensus.cc:998] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05: Attempting to remove follower 60e8857abe7c41038af46ebb5a095a90 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 60e8857abe7c41038af46ebb5a095a90 for more than 300 seconds (300.426s)
I20240501 08:05:09.822906 31786 consensus_queue.cc:237] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:05:09.824925 11401 raft_consensus.cc:1261] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:05:09.825379 32265 consensus_queue.cc:1035] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:05:09.826377 13185 raft_consensus.cc:1261] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:05:09.828660 32388 consensus_queue.cc:1035] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:05:09.831017 32388 raft_consensus.cc:2932] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:09.834014 11401 raft_consensus.cc:2932] T 17bb1ff829bc47a6a0bd6030517652aa P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:09.834362 11123 catalog_manager.cc:5484] T 17bb1ff829bc47a6a0bd6030517652aa P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index 7 to 8, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New cstate: current_term: 2 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:09.858734 13185 raft_consensus.cc:2932] T 17bb1ff829bc47a6a0bd6030517652aa P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:09.863406 31786 raft_consensus.cc:998] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05: Attempting to remove follower 60e8857abe7c41038af46ebb5a095a90 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 60e8857abe7c41038af46ebb5a095a90 for more than 300 seconds (300.083s)
I20240501 08:05:09.863780 31786 consensus_queue.cc:237] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 08:05:09.865509 11401 raft_consensus.cc:1261] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:05:09.865936 32388 consensus_queue.cc:1035] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:05:09.868311 32388 raft_consensus.cc:2932] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 08:05:09.871728 11123 catalog_manager.cc:5484] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index 5 to 6, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 08:05:09.872500 11401 raft_consensus.cc:2932] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 08:05:09.881171 11268 consensus_queue.cc:237] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:05:09.882400 11401 raft_consensus.cc:1261] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 08:05:09.883642 11206 consensus_peers.cc:485] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a266f266c3d4fd49fa8b325f1045d61. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:09.883997 32211 raft_consensus.cc:998] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2: Attempting to remove follower e5c03261d31c4d9a91077a17e1cd0336 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5c03261d31c4d9a91077a17e1cd0336 for more than 300 seconds (300.024s)
I20240501 08:05:09.884377 32211 consensus_queue.cc:237] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:05:09.886301 13448 raft_consensus.cc:1261] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:05:09.886493 32388 consensus_queue.cc:1035] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:05:09.887562 32211 consensus_queue.cc:1035] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:05:09.889411 32388 raft_consensus.cc:2932] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:09.889825 32211 raft_consensus.cc:2932] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:09.890587 13448 raft_consensus.cc:2932] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:09.891202 11113 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 7a266f266c3d4fd49fa8b325f1045d61 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:05:09.891747 11401 raft_consensus.cc:2932] T 7a266f266c3d4fd49fa8b325f1045d61 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:09.893091 11123 catalog_manager.cc:5484] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 reported cstate change: config changed from index 8 to 9, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New cstate: current_term: 2 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:09.893066 11125 catalog_manager.cc:5484] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index 6 to 7, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:09.901623 11661 consensus_queue.cc:237] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 08:05:09.903498 13448 raft_consensus.cc:1261] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:05:09.904268 32490 consensus_queue.cc:1035] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:05:09.907735 32490 raft_consensus.cc:2932] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 08:05:09.910707 11112 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 867ba08a15b44d4c9e6fa92b0d518d1d with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:05:09.911114 13448 raft_consensus.cc:2932] T 867ba08a15b44d4c9e6fa92b0d518d1d P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 08:05:09.912236 11120 catalog_manager.cc:5484] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 reported cstate change: config changed from index 9 to 10, VOTER 655ebda1fb2c49c4af7087a760529835 (127.0.99.130) added. New cstate: current_term: 2 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
W20240501 08:05:09.913408 11596 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575): Couldn't send request to peer 655ebda1fb2c49c4af7087a760529835. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 867ba08a15b44d4c9e6fa92b0d518d1d. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:09.927600 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:09.930518 30794 raft_consensus.cc:998] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6: Attempting to remove follower 60e8857abe7c41038af46ebb5a095a90 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 60e8857abe7c41038af46ebb5a095a90 for more than 300 seconds (300.175s)
I20240501 08:05:09.930948 30794 consensus_queue.cc:237] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:05:09.932853 11268 raft_consensus.cc:1261] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:05:09.933476 32445 consensus_queue.cc:1035] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:05:09.935302 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:09.942107 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:09.942445 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:09.944535 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:09.950655 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:09.970108 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:09.970158 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:09.974149 32445 raft_consensus.cc:2932] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:05:09.976747 11268 raft_consensus.cc:2932] T 6d214be86534458091f54a386d7d3334 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:05:09.976967 11119 catalog_manager.cc:5484] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
W20240501 08:05:10.040603 11467 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 08:05:10.049266 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 08:05:10.050621 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:10.053021 32527 raft_consensus.cc:998] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6: Attempting to remove follower 60e8857abe7c41038af46ebb5a095a90 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 60e8857abe7c41038af46ebb5a095a90 for more than 300 seconds (300.189s)
I20240501 08:05:10.053184 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:10.053450 32527 consensus_queue.cc:237] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:05:10.055189 11401 raft_consensus.cc:1261] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:05:10.055773 30794 consensus_queue.cc:1035] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:05:10.056164 30794 raft_consensus.cc:998] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6: Attempting to remove follower 60e8857abe7c41038af46ebb5a095a90 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 60e8857abe7c41038af46ebb5a095a90 for more than 300 seconds (300.140s)
I20240501 08:05:10.056519 30794 consensus_queue.cc:237] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:05:10.060355 13449 raft_consensus.cc:1261] T b4707b952d174a9bb877ad8bd7a11268 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:05:10.061244 32527 consensus_queue.cc:1035] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:05:10.065037 11661 raft_consensus.cc:1261] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
W20240501 08:05:10.065021 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:10.065485 32527 consensus_queue.cc:1035] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:05:10.067284 32527 raft_consensus.cc:2932] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:10.070173 13449 raft_consensus.cc:2932] T b4707b952d174a9bb877ad8bd7a11268 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:10.071004 11121 catalog_manager.cc:5484] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:10.071033 11401 raft_consensus.cc:2932] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:10.078677 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:10.090087 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:10.093791 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:10.094653 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:10.096367 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:10.105816 5411 raft_consensus.cc:998] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9: Attempting to remove follower 60e8857abe7c41038af46ebb5a095a90 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 60e8857abe7c41038af46ebb5a095a90 for more than 300 seconds (300.113s)
I20240501 08:05:10.106245 5411 consensus_queue.cc:237] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 08:05:10.108237 11268 raft_consensus.cc:1261] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Refusing update from remote peer 53d1d2c0767b4f6195b69758a3dd6cf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 08:05:10.109139 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:10.109962 32265 raft_consensus.cc:998] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05: Attempting to remove follower 60e8857abe7c41038af46ebb5a095a90 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 60e8857abe7c41038af46ebb5a095a90 for more than 300 seconds (300.301s)
I20240501 08:05:10.110128 5411 consensus_queue.cc:1035] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 08:05:10.110299 32265 consensus_queue.cc:237] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 08:05:10.111652 11401 raft_consensus.cc:1261] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:05:10.112046 31786 consensus_queue.cc:1035] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:05:10.112709 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:10.115967 31786 raft_consensus.cc:998] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05: Attempting to remove follower 60e8857abe7c41038af46ebb5a095a90 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 60e8857abe7c41038af46ebb5a095a90 for more than 300 seconds (300.140s)
I20240501 08:05:10.116283 31786 consensus_queue.cc:237] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:05:10.119791 13449 raft_consensus.cc:1261] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:05:10.119899 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:10.120203 32388 consensus_queue.cc:1035] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:05:10.123677 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:10.131671 11336 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:05:10.135366 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:10.142324 32388 raft_consensus.cc:2932] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 08:05:10.145380 11401 raft_consensus.cc:2932] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 08:05:10.145385 11121 catalog_manager.cc:5484] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New cstate: current_term: 2 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 08:05:10.150492 11269 consensus_queue.cc:237] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:05:10.152427 11401 raft_consensus.cc:1261] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:05:10.152947 32265 consensus_queue.cc:1035] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 08:05:10.156339 11203 consensus_peers.cc:485] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): Couldn't send request to peer 603919bb02364c04b6c530245b8b02f2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:10.165685 32265 raft_consensus.cc:2932] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:05:10.168496 31763 raft_consensus.cc:998] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835: Attempting to remove follower 60e8857abe7c41038af46ebb5a095a90 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 60e8857abe7c41038af46ebb5a095a90 for more than 300 seconds (300.164s)
I20240501 08:05:10.168864 31763 consensus_queue.cc:237] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:05:10.169903 11113 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet b21b6e2023e842739a7c0cc946e7c2fd with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:05:10.170544 11661 raft_consensus.cc:1261] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:05:10.170861 11401 raft_consensus.cc:2932] T b21b6e2023e842739a7c0cc946e7c2fd P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:05:10.170931 11121 catalog_manager.cc:5484] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index 7 to 8, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New cstate: current_term: 2 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:05:10.171381 13449 raft_consensus.cc:1261] T 3f6e4ae97e884219ac7d0d94997d9cc9 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:05:10.171784 31763 consensus_queue.cc:1035] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:05:10.172874 31763 consensus_queue.cc:1035] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:05:10.175256 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:10.179896 32527 raft_consensus.cc:2932] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:05:10.182884 11663 raft_consensus.cc:2932] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:05:10.188889 11119 catalog_manager.cc:5484] T bd8348990f1b496bb62d8c0484193442 P 603919bb02364c04b6c530245b8b02f2 reported cstate change: config changed from index 9 to 10, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:05:10.195928 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:10.201340 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:10.203902 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:10.211153 32639 raft_consensus.cc:2932] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:10.214176 11121 catalog_manager.cc:5484] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:10.214833 11661 raft_consensus.cc:2932] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:10.216859 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:10.220115 32388 raft_consensus.cc:2932] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:10.222951 13448 raft_consensus.cc:2932] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:10.223538 32629 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:05:10.224035 11268 raft_consensus.cc:2932] T 25cee7e51c7c438aa721d71fb82dc71e P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:05:10.225865 11119 catalog_manager.cc:5484] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608 reported cstate change: config changed from index 8 to 9, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New cstate: current_term: 2 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:10.226593 11123 catalog_manager.cc:5484] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 reported cstate change: config changed from index 11 to 12, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New cstate: current_term: 2 leader_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:05:10.228688 13448 raft_consensus.cc:2932] T 3f6e4ae97e884219ac7d0d94997d9cc9 P ca177138f2c44d2eafa1646b795f7608 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:10.232066 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:05:10.232426 11268 consensus_queue.cc:237] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
W20240501 08:05:10.234766 11203 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771): Couldn't send request to peer 53d1d2c0767b4f6195b69758a3dd6cf9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9431cf5853c4148ac38855b1c083d4f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:10.237043 13448 raft_consensus.cc:1261] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:05:10.237527 32265 consensus_queue.cc:1035] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:05:10.239622 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:10.251148 32265 raft_consensus.cc:2932] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:10.252892 11113 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet b9431cf5853c4148ac38855b1c083d4f with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:05:10.253557 32211 raft_consensus.cc:998] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2: Attempting to remove follower 60e8857abe7c41038af46ebb5a095a90 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 60e8857abe7c41038af46ebb5a095a90 for more than 300 seconds (300.285s)
I20240501 08:05:10.253957 32211 consensus_queue.cc:237] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 08:05:10.254411 11121 catalog_manager.cc:5484] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index 9 to 10, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New cstate: current_term: 2 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:10.256800 11268 raft_consensus.cc:1261] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:05:10.257282 32490 consensus_queue.cc:1035] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:05:10.258512 11531 raft_consensus.cc:1261] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:05:10.260522 32490 consensus_queue.cc:1035] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:05:10.262784 13449 raft_consensus.cc:2932] T b9431cf5853c4148ac38855b1c083d4f P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:10.265221 32490 raft_consensus.cc:2932] T ed88ea311f474ab28cc6fbe6069258bf P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:05:10.267740 11268 raft_consensus.cc:2932] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:05:10.273471 11123 catalog_manager.cc:5484] T ed88ea311f474ab28cc6fbe6069258bf P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:05:10.279047 11531 raft_consensus.cc:2932] T ed88ea311f474ab28cc6fbe6069258bf P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:05:10.331993 32639 raft_consensus.cc:998] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835: Attempting to remove follower 60e8857abe7c41038af46ebb5a095a90 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 60e8857abe7c41038af46ebb5a095a90 for more than 300 seconds (300.390s)
I20240501 08:05:10.332401 32639 consensus_queue.cc:237] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:05:10.334116 11661 raft_consensus.cc:1261] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:05:10.334899 32637 consensus_queue.cc:1035] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:05:10.335136 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:10.337319 32637 raft_consensus.cc:2932] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:05:10.340675 11121 catalog_manager.cc:5484] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:05:10.341488 11661 raft_consensus.cc:2932] T 64613ea537c842f29d810e829278394e P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } } }
I20240501 08:05:10.360188 32734 ts_tablet_manager.cc:902] T 7a266f266c3d4fd49fa8b325f1045d61 P ca177138f2c44d2eafa1646b795f7608: Initiating tablet copy from peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
F20240501 08:05:10.360473 32734 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-164fd265561e47128e7ff585b8cfab07.queue_time_us, tablet-download-pool-164fd265561e47128e7ff585b8cfab07.queue_time_us) (tablet-download-pool-164fd265561e47128e7ff585b8cfab07.run_cpu_time_us, tablet-download-pool-164fd265561e47128e7ff585b8cfab07.run_cpu_time_us) (tablet-download-pool-164fd265561e47128e7ff585b8cfab07.run_wall_time_us, tablet-download-pool-164fd265561e47128e7ff585b8cfab07.run_wall_time_us) (tablet-download-pool-3f6e4ae97e884219ac7d0d94997d9cc9.queue_time_us, tablet-download-pool-3f6e4ae97e884219ac7d0d94997d9cc9.queue_time_us) (tablet-download-pool-3f6e4ae97e884219ac7d0d94997d9cc9.run_cpu_time_us, tablet-download-pool-3f6e4ae97e884219ac7d0d94997d9cc9.run_cpu_time_us) (tablet-download-pool-3f6e4ae97e884219ac7d0d94997d9cc9.run_wall_time_us, tablet-download-pool-3f6e4ae97e884219ac7d0d94997d9cc9.run_wall_time_us) (tablet-download-pool-53193305f868433c852755e8a8a81766.queue_time_us, tablet-download-pool-53193305f868433c852755e8a8a81766.queue_time_us) (tablet-download-pool-53193305f868433c852755e8a8a81766.run_cpu_time_us, tablet-download-pool-53193305f868433c852755e8a8a81766.run_cpu_time_us) (tablet-download-pool-53193305f868433c852755e8a8a81766.run_wall_time_us, tablet-download-pool-53193305f868433c852755e8a8a81766.run_wall_time_us) (tablet-download-pool-5b7de08d2f004cd8abc2bbe1cf7b1444.queue_time_us, tablet-download-pool-5b7de08d2f004cd8abc2bbe1cf7b1444.queue_time_us) (tablet-download-pool-5b7de08d2f004cd8abc2bbe1cf7b1444.run_cpu_time_us, tablet-download-pool-5b7de08d2f004cd8abc2bbe1cf7b1444.run_cpu_time_us) (tablet-download-pool-5b7de08d2f004cd8abc2bbe1cf7b1444.run_wall_time_us, tablet-download-pool-5b7de08d2f004cd8abc2bbe1cf7b1444.run_wall_time_us) (tablet-download-pool-5c47aaeb16c945ae84ece505aada2a7c.queue_time_us, tablet-download-pool-5c47aaeb16c945ae84ece505aada2a7c.queue_time_us) (tablet-download-pool-5c47aaeb16c945ae84ece505aada2a7c.run_cpu_time_us, tablet-download-pool-5c47aaeb16c945ae84ece505aada2a7c.run_cpu_time_us) (tablet-download-pool-5c47aaeb16c945ae84ece505aada2a7c.run_wall_time_us, tablet-download-pool-5c47aaeb16c945ae84ece505aada2a7c.run_wall_time_us) (tablet-download-pool-5e8550ea95984d08937b9b3e2af12bec.queue_time_us, tablet-download-pool-5e8550ea95984d08937b9b3e2af12bec.queue_time_us) (tablet-download-pool-5e8550ea95984d08937b9b3e2af12bec.run_cpu_time_us, tablet-download-pool-5e8550ea95984d08937b9b3e2af12bec.run_cpu_time_us) (tablet-download-pool-5e8550ea95984d08937b9b3e2af12bec.run_wall_time_us, tablet-download-pool-5e8550ea95984d08937b9b3e2af12bec.run_wall_time_us) (tablet-download-pool-7a266f266c3d4fd49fa8b325f1045d61.queue_time_us, tablet-download-pool-7a266f266c3d4fd49fa8b325f1045d61.queue_time_us) (tablet-download-pool-867ba08a15b44d4c9e6fa92b0d518d1d.queue_time_us, tablet-download-pool-867ba08a15b44d4c9e6fa92b0d518d1d.queue_time_us) (tablet-download-pool-867ba08a15b44d4c9e6fa92b0d518d1d.run_cpu_time_us, tablet-download-pool-867ba08a15b44d4c9e6fa92b0d518d1d.run_cpu_time_us) (tablet-download-pool-867ba08a15b44d4c9e6fa92b0d518d1d.run_wall_time_us, tablet-download-pool-867ba08a15b44d4c9e6fa92b0d518d1d.run_wall_time_us) (tablet-download-pool-90f11b5693d7456d8a604e6c8726e69a.queue_time_us, tablet-download-pool-90f11b5693d7456d8a604e6c8726e69a.queue_time_us) (tablet-download-pool-90f11b5693d7456d8a604e6c8726e69a.run_cpu_time_us, tablet-download-pool-90f11b5693d7456d8a604e6c8726e69a.run_cpu_time_us) (tablet-download-pool-90f11b5693d7456d8a604e6c8726e69a.run_wall_time_us, tablet-download-pool-90f11b5693d7456d8a604e6c8726e69a.run_wall_time_us) (tablet-download-pool-a24be92aec7544089c7818f2802b9c85.queue_time_us, tablet-download-pool-a24be92aec7544089c7818f2802b9c85.queue_time_us) (tablet-download-pool-a24be92aec7544089c7818f2802b9c85.run_cpu_time_us, tablet-download-pool-a24be92aec7544089c7818f2802b9c85.run_cpu_time_us) (tablet-download-pool-a24be92aec7544089c7818f2802b9c85.run_wall_time_us, tablet-download-pool-a24be92aec7544089c7818f2802b9c85.run_wall_time_us) (tablet-download-pool-b4707b952d174a9bb877ad8bd7a11268.queue_time_us, tablet-download-pool-b4707b952d174a9bb877ad8bd7a11268.queue_time_us) (tablet-download-pool-b4707b952d174a9bb877ad8bd7a11268.run_cpu_time_us, tablet-download-pool-b4707b952d174a9bb877ad8bd7a11268.run_cpu_time_us) (tablet-download-pool-b4707b952d174a9bb877ad8bd7a11268.run_wall_time_us, tablet-download-pool-b4707b952d174a9bb877ad8bd7a11268.run_wall_time_us) (tablet-download-pool-b9431cf5853c4148ac38855b1c083d4f.queue_time_us, tablet-download-pool-b9431cf5853c4148ac38855b1c083d4f.queue_time_us) (tablet-download-pool-b9431cf5853c4148ac38855b1c083d4f.run_cpu_time_us, tablet-download-pool-b9431cf5853c4148ac38855b1c083d4f.run_cpu_time_us) (tablet-download-pool-b9431cf5853c4148ac38855b1c083d4f.run_wall_time_us, tablet-download-pool-b9431cf5853c4148ac38855b1c083d4f.run_wall_time_us) (tablet-download-pool-c95c4f3ef08f44d1b69fb7b98426f5af.queue_time_us, tablet-download-pool-c95c4f3ef08f44d1b69fb7b98426f5af.queue_time_us) (tablet-download-pool-c95c4f3ef08f44d1b69fb7b98426f5af.run_cpu_time_us, tablet-download-pool-c95c4f3ef08f44d1b69fb7b98426f5af.run_cpu_time_us) (tablet-download-pool-c95c4f3ef08f44d1b69fb7b98426f5af.run_wall_time_us, tablet-download-pool-c95c4f3ef08f44d1b69fb7b98426f5af.run_wall_time_us) (tablet-download-pool-f2e63d24d7df4fa3b2b26695e70ebe9f.queue_time_us, tablet-download-pool-f2e63d24d7df4fa3b2b26695e70ebe9f.queue_time_us) (tablet-download-pool-f2e63d24d7df4fa3b2b26695e70ebe9f.run_cpu_time_us, tablet-download-pool-f2e63d24d7df4fa3b2b26695e70ebe9f.run_cpu_time_us) (tablet-download-pool-f2e63d24d7df4fa3b2b26695e70ebe9f.run_wall_time_us, tablet-download-pool-f2e63d24d7df4fa3b2b26695e70ebe9f.run_wall_time_us) (tablet-download-pool-f80db2e37bbc4716ad3a5200c61ee97b.queue_time_us, tablet-download-pool-f80db2e37bbc4716ad3a5200c61ee97b.queue_time_us) (tablet-download-pool-f80db2e37bbc4716ad3a5200c61ee97b.run_cpu_time_us, tablet-download-pool-f80db2e37bbc4716ad3a5200c61ee97b.run_cpu_time_us) (tablet-download-pool-f80db2e37bbc4716ad3a5200c61ee97b.run_wall_time_us, tablet-download-pool-f80db2e37bbc4716ad3a5200c61ee97b.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240501 08:05:10.369940 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:10.380589 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
@ 0x7f40c49a6dcd google::LogMessage::Fail() at ??:0
@ 0x7f40c49aab93 google::LogMessage::SendToLog() at ??:0
I20240501 08:05:10.385360 31763 raft_consensus.cc:998] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835: Attempting to remove follower e5c03261d31c4d9a91077a17e1cd0336 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5c03261d31c4d9a91077a17e1cd0336 for more than 300 seconds (300.252s)
I20240501 08:05:10.385710 31763 consensus_queue.cc:237] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:05:10.386945 13185 raft_consensus.cc:1261] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:05:10.387452 32639 consensus_queue.cc:1035] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:05:10.389525 32639 raft_consensus.cc:2932] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:10.391006 13185 raft_consensus.cc:2932] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:10.393222 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:10.394213 32445 raft_consensus.cc:998] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6: Attempting to remove follower e5c03261d31c4d9a91077a17e1cd0336 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5c03261d31c4d9a91077a17e1cd0336 for more than 300 seconds (300.255s)
I20240501 08:05:10.394555 32445 consensus_queue.cc:237] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:05:10.394515 11128 catalog_manager.cc:5484] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9 reported cstate change: config changed from index 7 to 8, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
@ 0x7f40c49a67cc google::LogMessage::Flush() at ??:0
I20240501 08:05:10.412586 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
@ 0x7f40c49a7f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 08:05:10.424461 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:10.425339 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:10.425976 11401 consensus_queue.cc:237] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:05:10.426445 13185 raft_consensus.cc:1261] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:05:10.426921 32527 consensus_queue.cc:1035] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 08:05:10.427940 11334 consensus_peers.cc:485] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 -> Peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): Couldn't send request to peer 603919bb02364c04b6c530245b8b02f2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:10.427968 13186 raft_consensus.cc:1261] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:05:10.429234 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:10.429189 32637 consensus_queue.cc:1035] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:05:10.430917 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:10.432672 32756 ts_tablet_manager.cc:902] T 867ba08a15b44d4c9e6fa92b0d518d1d P 655ebda1fb2c49c4af7087a760529835: Initiating tablet copy from peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 08:05:10.434763 13429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
@ 0x7f40c5e10044 kudu::TraceMetrics::InternName() at ??:0
I20240501 08:05:10.439961 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:10.440907 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:10.459703 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
@ 0x7f40c5df4f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f40c5df2efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f40c89a577a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 08:05:10.483362 31763 raft_consensus.cc:2932] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
@ 0x7f40c89aea45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 08:05:10.483892 13186 raft_consensus.cc:2932] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:05:10.487363 11128 catalog_manager.cc:5484] T 7c23e3bd7ec34876956046e5d32e4d82 P 53d1d2c0767b4f6195b69758a3dd6cf9 reported cstate change: config changed from index 8 to 9, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
@ 0x7f40c8a2f76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 08:05:10.490185 11112 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 7c23e3bd7ec34876956046e5d32e4d82 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
@ 0x7f40c8a434f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f40c8a2e226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f40c8a3dc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55d0b67f375c std::function<>::operator()() at ??:0
@ 0x7f40c5df7e06 kudu::ThreadPool::DispatchThread() at ??:0
I20240501 08:05:10.501169 32527 raft_consensus.cc:998] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6: Attempting to remove follower e5c03261d31c4d9a91077a17e1cd0336 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5c03261d31c4d9a91077a17e1cd0336 for more than 300 seconds (300.245s)
I20240501 08:05:10.501514 32527 consensus_queue.cc:237] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
@ 0x7f40c5df86ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240501 08:05:10.503122 13449 raft_consensus.cc:1261] T 90f11b5693d7456d8a604e6c8726e69a P ca177138f2c44d2eafa1646b795f7608 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 08:05:10.503947 32445 consensus_queue.cc:1035] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
@ 0x7f40c5dfa03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55d0b67f375c std::function<>::operator()() at ??:0
@ 0x7f40c5de7807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f40c66256db start_thread at ??:0
@ 0x7f40c3b5d71f clone at ??:0
I20240501 08:05:10.515347 32756 tablet_copy_client.cc:308] T 867ba08a15b44d4c9e6fa92b0d518d1d P 655ebda1fb2c49c4af7087a760529835: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:44403
I20240501 08:05:10.518200 11683 tablet_copy_service.cc:140] P 603919bb02364c04b6c530245b8b02f2: Received BeginTabletCopySession request for tablet 867ba08a15b44d4c9e6fa92b0d518d1d from peer 655ebda1fb2c49c4af7087a760529835 ({username='slave'} at 127.0.99.130:59463)
I20240501 08:05:10.518317 11683 tablet_copy_service.cc:161] P 603919bb02364c04b6c530245b8b02f2: Beginning new tablet copy session on tablet 867ba08a15b44d4c9e6fa92b0d518d1d from peer 655ebda1fb2c49c4af7087a760529835 at {username='slave'} at 127.0.99.130:59463: session id = 655ebda1fb2c49c4af7087a760529835-867ba08a15b44d4c9e6fa92b0d518d1d
I20240501 08:05:10.519743 11683 tablet_copy_source_session.cc:205] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:05:10.520521 30794 raft_consensus.cc:2932] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:10.521031 13185 raft_consensus.cc:2932] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:10.523465 32756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 867ba08a15b44d4c9e6fa92b0d518d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:10.523396 11119 catalog_manager.cc:5484] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 7 to 8, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:10.525831 32756 tablet_copy_client.cc:783] T 867ba08a15b44d4c9e6fa92b0d518d1d P 655ebda1fb2c49c4af7087a760529835: tablet copy: Starting download of 5 data blocks...
I20240501 08:05:10.530791 32660 raft_consensus.cc:998] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2: Attempting to remove follower e5c03261d31c4d9a91077a17e1cd0336 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5c03261d31c4d9a91077a17e1cd0336 for more than 300 seconds (300.215s)
I20240501 08:05:10.531239 32660 consensus_queue.cc:237] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:05:10.533541 11531 consensus_queue.cc:237] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:05:10.535091 13185 raft_consensus.cc:1261] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 08:05:10.535419 11465 consensus_peers.cc:485] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): Couldn't send request to peer 603919bb02364c04b6c530245b8b02f2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:10.535604 32445 consensus_queue.cc:1035] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:05:10.537518 31763 raft_consensus.cc:998] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835: Attempting to remove follower e5c03261d31c4d9a91077a17e1cd0336 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5c03261d31c4d9a91077a17e1cd0336 for more than 300 seconds (300.411s)
I20240501 08:05:10.537883 31763 consensus_queue.cc:237] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } }
I20240501 08:05:10.539451 32445 raft_consensus.cc:2932] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:05:10.541220 13185 raft_consensus.cc:1261] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:05:10.541648 32211 consensus_queue.cc:1035] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:05:10.541914 11114 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 48b0ff0769dc426190f038811155c741 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:05:10.542766 13186 raft_consensus.cc:2932] T 48b0ff0769dc426190f038811155c741 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:05:10.546597 11128 catalog_manager.cc:5484] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 8 to 9, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:05:10.548777 32490 raft_consensus.cc:2932] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:10.551244 13185 raft_consensus.cc:2932] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:10.554607 11119 catalog_manager.cc:5484] T f80db2e37bbc4716ad3a5200c61ee97b P 53d1d2c0767b4f6195b69758a3dd6cf9 reported cstate change: config changed from index 8 to 9, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:05:10.563869 32756 tablet_copy_client.cc:647] T 867ba08a15b44d4c9e6fa92b0d518d1d P 655ebda1fb2c49c4af7087a760529835: tablet copy: Starting download of 1 WAL segments...
I20240501 08:05:10.565541 32756 tablet_copy_client.cc:523] T 867ba08a15b44d4c9e6fa92b0d518d1d P 655ebda1fb2c49c4af7087a760529835: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:05:10.567544 32756 tablet_bootstrap.cc:492] T 867ba08a15b44d4c9e6fa92b0d518d1d P 655ebda1fb2c49c4af7087a760529835: Bootstrap starting.
I20240501 08:05:10.571368 339 ts_tablet_manager.cc:902] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: Initiating tablet copy from peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 08:05:10.571946 339 tablet_copy_client.cc:308] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46085
I20240501 08:05:10.572453 11290 tablet_copy_service.cc:140] P e2dafc1866314669ad0de8fb7bf7fd05: Received BeginTabletCopySession request for tablet b21b6e2023e842739a7c0cc946e7c2fd from peer 603919bb02364c04b6c530245b8b02f2 ({username='slave'} at 127.0.99.132:38941)
I20240501 08:05:10.572564 11290 tablet_copy_service.cc:161] P e2dafc1866314669ad0de8fb7bf7fd05: Beginning new tablet copy session on tablet b21b6e2023e842739a7c0cc946e7c2fd from peer 603919bb02364c04b6c530245b8b02f2 at {username='slave'} at 127.0.99.132:38941: session id = 603919bb02364c04b6c530245b8b02f2-b21b6e2023e842739a7c0cc946e7c2fd
I20240501 08:05:10.574085 11290 tablet_copy_source_session.cc:205] T b21b6e2023e842739a7c0cc946e7c2fd P e2dafc1866314669ad0de8fb7bf7fd05: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:05:10.575912 339 ts_tablet_manager.cc:1875] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:05:10.579161 339 ts_tablet_manager.cc:1888] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.5
I20240501 08:05:10.579397 339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21b6e2023e842739a7c0cc946e7c2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:10.580878 339 tablet_copy_client.cc:783] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: tablet copy: Starting download of 5 data blocks...
I20240501 08:05:10.581130 32756 tablet_bootstrap.cc:492] T 867ba08a15b44d4c9e6fa92b0d518d1d P 655ebda1fb2c49c4af7087a760529835: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:10.581461 32756 tablet_bootstrap.cc:492] T 867ba08a15b44d4c9e6fa92b0d518d1d P 655ebda1fb2c49c4af7087a760529835: Bootstrap complete.
I20240501 08:05:10.581658 32756 ts_tablet_manager.cc:1367] T 867ba08a15b44d4c9e6fa92b0d518d1d P 655ebda1fb2c49c4af7087a760529835: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20240501 08:05:10.582135 32756 raft_consensus.cc:348] T 867ba08a15b44d4c9e6fa92b0d518d1d P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 08:05:10.582242 32756 raft_consensus.cc:729] T 867ba08a15b44d4c9e6fa92b0d518d1d P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655ebda1fb2c49c4af7087a760529835, State: Initialized, Role: FOLLOWER
I20240501 08:05:10.582388 32756 consensus_queue.cc:260] T 867ba08a15b44d4c9e6fa92b0d518d1d P 655ebda1fb2c49c4af7087a760529835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 08:05:10.589284 32756 ts_tablet_manager.cc:1398] T 867ba08a15b44d4c9e6fa92b0d518d1d P 655ebda1fb2c49c4af7087a760529835: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20240501 08:05:10.591603 11683 tablet_copy_service.cc:342] P 603919bb02364c04b6c530245b8b02f2: Request end of tablet copy session 655ebda1fb2c49c4af7087a760529835-867ba08a15b44d4c9e6fa92b0d518d1d received from {username='slave'} at 127.0.99.130:59463
I20240501 08:05:10.591711 11683 tablet_copy_service.cc:433] P 603919bb02364c04b6c530245b8b02f2: ending tablet copy session 655ebda1fb2c49c4af7087a760529835-867ba08a15b44d4c9e6fa92b0d518d1d on tablet 867ba08a15b44d4c9e6fa92b0d518d1d with peer 655ebda1fb2c49c4af7087a760529835
I20240501 08:05:10.591732 339 tablet_copy_client.cc:647] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:05:10.593011 339 tablet_copy_client.cc:523] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:05:10.594393 339 tablet_bootstrap.cc:492] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 08:05:10.613579 339 tablet_bootstrap.cc:492] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:10.618064 339 tablet_bootstrap.cc:492] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: Bootstrap complete.
I20240501 08:05:10.618371 339 ts_tablet_manager.cc:1367] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.000s
I20240501 08:05:10.618989 339 raft_consensus.cc:348] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:05:10.619125 339 raft_consensus.cc:729] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 08:05:10.619324 339 consensus_queue.cc:260] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:05:10.622017 339 ts_tablet_manager.cc:1398] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 08:05:10.624507 11290 tablet_copy_service.cc:342] P e2dafc1866314669ad0de8fb7bf7fd05: Request end of tablet copy session 603919bb02364c04b6c530245b8b02f2-b21b6e2023e842739a7c0cc946e7c2fd received from {username='slave'} at 127.0.99.132:38941
I20240501 08:05:10.624600 11290 tablet_copy_service.cc:433] P e2dafc1866314669ad0de8fb7bf7fd05: ending tablet copy session 603919bb02364c04b6c530245b8b02f2-b21b6e2023e842739a7c0cc946e7c2fd on tablet b21b6e2023e842739a7c0cc946e7c2fd with peer 603919bb02364c04b6c530245b8b02f2
I20240501 08:05:10.638098 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:10.644795 11448 maintenance_manager.cc:419] P 655ebda1fb2c49c4af7087a760529835: Scheduling UndoDeltaBlockGCOp(867ba08a15b44d4c9e6fa92b0d518d1d): 4583 bytes on disk
I20240501 08:05:10.645293 11338 maintenance_manager.cc:643] P 655ebda1fb2c49c4af7087a760529835: UndoDeltaBlockGCOp(867ba08a15b44d4c9e6fa92b0d518d1d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240501 08:05:10.649989 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:10.657431 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:10.657860 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:10.660416 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling UndoDeltaBlockGCOp(b21b6e2023e842739a7c0cc946e7c2fd): 4839 bytes on disk
I20240501 08:05:10.660977 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: UndoDeltaBlockGCOp(b21b6e2023e842739a7c0cc946e7c2fd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240501 08:05:10.671176 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:10.673054 32637 raft_consensus.cc:998] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835: Attempting to remove follower 60e8857abe7c41038af46ebb5a095a90 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 60e8857abe7c41038af46ebb5a095a90 for more than 300 seconds (300.402s)
I20240501 08:05:10.673411 32637 consensus_queue.cc:237] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 08:05:10.675662 11531 raft_consensus.cc:1261] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer 655ebda1fb2c49c4af7087a760529835: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:05:10.676282 31763 consensus_queue.cc:1035] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:05:10.688338 32639 raft_consensus.cc:2932] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 08:05:10.688802 11531 raft_consensus.cc:2932] T 164fd265561e47128e7ff585b8cfab07 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 08:05:10.691663 11126 catalog_manager.cc:5484] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 8 to 9, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "655ebda1fb2c49c4af7087a760529835" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 08:05:10.730947 32388 raft_consensus.cc:998] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05: Attempting to remove follower e5c03261d31c4d9a91077a17e1cd0336 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5c03261d31c4d9a91077a17e1cd0336 for more than 300 seconds (300.382s)
I20240501 08:05:10.731384 32388 consensus_queue.cc:237] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:05:10.733274 11400 raft_consensus.cc:1261] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:05:10.733395 13185 raft_consensus.cc:1261] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:05:10.733839 31786 consensus_queue.cc:1035] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:05:10.734000 32265 consensus_queue.cc:1035] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:05:10.736411 31786 raft_consensus.cc:2932] T ea97f34590684bcf998228245dbcd9b0 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:10.738525 13185 raft_consensus.cc:2932] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:10.741766 11121 catalog_manager.cc:5484] T ea97f34590684bcf998228245dbcd9b0 P 53d1d2c0767b4f6195b69758a3dd6cf9 reported cstate change: config changed from index 6 to 7, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
W20240501 08:05:10.741770 11115 catalog_manager.cc:4573] TS e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f80db2e37bbc4716ad3a5200c61ee97b: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:10.751777 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:10.753715 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:10.759425 32388 raft_consensus.cc:998] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05: Attempting to remove follower e5c03261d31c4d9a91077a17e1cd0336 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5c03261d31c4d9a91077a17e1cd0336 for more than 300 seconds (300.475s)
I20240501 08:05:10.759821 32388 consensus_queue.cc:237] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } }
I20240501 08:05:10.761660 11531 raft_consensus.cc:1261] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:05:10.762732 32388 consensus_queue.cc:1035] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:05:10.765610 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:10.768251 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:10.768885 11400 raft_consensus.cc:2932] T ea97f34590684bcf998228245dbcd9b0 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:10.791272 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:10.795882 31786 raft_consensus.cc:2932] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 08:05:10.798012 11531 raft_consensus.cc:2932] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 08:05:10.801208 11126 catalog_manager.cc:5484] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 reported cstate change: config changed from index 5 to 6, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } }
I20240501 08:05:10.805104 11268 consensus_queue.cc:237] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:05:10.806442 11531 raft_consensus.cc:1261] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Refusing update from remote peer e2dafc1866314669ad0de8fb7bf7fd05: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:05:10.806957 32265 consensus_queue.cc:1035] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:05:10.808842 32265 raft_consensus.cc:2932] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:10.810820 11113 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 18b146cc4e164e1d9f9f827217c9b3ab with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:05:10.810986 11400 raft_consensus.cc:1203] T 867ba08a15b44d4c9e6fa92b0d518d1d P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240501 08:05:10.811717 11531 raft_consensus.cc:2932] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
W20240501 08:05:10.814145 11203 consensus_peers.cc:485] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134:36771): Couldn't send request to peer 53d1d2c0767b4f6195b69758a3dd6cf9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18b146cc4e164e1d9f9f827217c9b3ab. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:10.815780 11121 catalog_manager.cc:5484] T 18b146cc4e164e1d9f9f827217c9b3ab P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 6 to 7, VOTER 53d1d2c0767b4f6195b69758a3dd6cf9 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:10.817413 429 ts_tablet_manager.cc:902] T b9431cf5853c4148ac38855b1c083d4f P 53d1d2c0767b4f6195b69758a3dd6cf9: Initiating tablet copy from peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 08:05:10.818006 429 tablet_copy_client.cc:308] T b9431cf5853c4148ac38855b1c083d4f P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46085
I20240501 08:05:10.822540 11290 tablet_copy_service.cc:140] P e2dafc1866314669ad0de8fb7bf7fd05: Received BeginTabletCopySession request for tablet b9431cf5853c4148ac38855b1c083d4f from peer 53d1d2c0767b4f6195b69758a3dd6cf9 ({username='slave'} at 127.0.99.134:37773)
I20240501 08:05:10.822695 11290 tablet_copy_service.cc:161] P e2dafc1866314669ad0de8fb7bf7fd05: Beginning new tablet copy session on tablet b9431cf5853c4148ac38855b1c083d4f from peer 53d1d2c0767b4f6195b69758a3dd6cf9 at {username='slave'} at 127.0.99.134:37773: session id = 53d1d2c0767b4f6195b69758a3dd6cf9-b9431cf5853c4148ac38855b1c083d4f
I20240501 08:05:10.823647 11290 tablet_copy_source_session.cc:205] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:05:10.824997 429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9431cf5853c4148ac38855b1c083d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:10.827441 429 tablet_copy_client.cc:783] T b9431cf5853c4148ac38855b1c083d4f P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 5 data blocks...
I20240501 08:05:10.832464 429 tablet_copy_client.cc:647] T b9431cf5853c4148ac38855b1c083d4f P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 1 WAL segments...
I20240501 08:05:10.833912 429 tablet_copy_client.cc:523] T b9431cf5853c4148ac38855b1c083d4f P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:05:10.835983 429 tablet_bootstrap.cc:492] T b9431cf5853c4148ac38855b1c083d4f P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap starting.
I20240501 08:05:10.846776 429 tablet_bootstrap.cc:492] T b9431cf5853c4148ac38855b1c083d4f P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:10.847076 429 tablet_bootstrap.cc:492] T b9431cf5853c4148ac38855b1c083d4f P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap complete.
I20240501 08:05:10.847295 429 ts_tablet_manager.cc:1367] T b9431cf5853c4148ac38855b1c083d4f P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240501 08:05:10.847728 429 raft_consensus.cc:348] T b9431cf5853c4148ac38855b1c083d4f P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:05:10.847832 429 raft_consensus.cc:729] T b9431cf5853c4148ac38855b1c083d4f P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d1d2c0767b4f6195b69758a3dd6cf9, State: Initialized, Role: FOLLOWER
I20240501 08:05:10.847959 429 consensus_queue.cc:260] T b9431cf5853c4148ac38855b1c083d4f P 53d1d2c0767b4f6195b69758a3dd6cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:05:10.848258 429 ts_tablet_manager.cc:1398] T b9431cf5853c4148ac38855b1c083d4f P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 08:05:10.848851 11290 tablet_copy_service.cc:342] P e2dafc1866314669ad0de8fb7bf7fd05: Request end of tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-b9431cf5853c4148ac38855b1c083d4f received from {username='slave'} at 127.0.99.134:37773
I20240501 08:05:10.848929 11290 tablet_copy_service.cc:433] P e2dafc1866314669ad0de8fb7bf7fd05: ending tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-b9431cf5853c4148ac38855b1c083d4f on tablet b9431cf5853c4148ac38855b1c083d4f with peer 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 08:05:10.862042 11468 connection.cc:535] client connection to 127.0.99.136:34503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:05:10.862135 11206 connection.cc:535] client connection to 127.0.99.136:34503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:05:10.862205 11337 connection.cc:535] client connection to 127.0.99.136:34503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:05:10.862689 11599 connection.cc:535] client connection to 127.0.99.136:34503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:05:10.863384 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:10.863461 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:10.863636 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:10.863700 11468 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:10.864782 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:10.864868 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:10.865211 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:10.865290 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:10.865324 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:05:10.872347 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:10.885569 13499 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskvJ2Dnc/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 07:59:53.931347 13508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:53.931610 13508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:53.934985 13508 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 07:59:55.043995 13508 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 07:59:55.651875 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6447707eb34f45d7a8d3704cd09d7786: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:55.652065 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 69f109081c7449be8819974c5363fa32: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:55.652182 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 98fb02bdb5f54b1699749e3c8d588a5e: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:55.652249 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af21d4c4fcb49a7ac0756a5ded5d57d: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:55.652333 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24b4f6e8cc8c44f6961591cac7beacb4: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:55.652474 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 69f109081c7449be8819974c5363fa32: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:55.652544 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:55.652637 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 701e50fb54354219a26221f76a7352d2: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:55.652714 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df5f9864bbf8499dbaa392bcb85701c0: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:55.652760 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 384b2b245c6b4931a4fc02c6b5d647b2: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:55.652823 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 36e20074d8dc4402bd7d9b2039213af5: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:55.652880 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 98fb02bdb5f54b1699749e3c8d588a5e: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:55.652930 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af21d4c4fcb49a7ac0756a5ded5d57d: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:55.652978 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 69f109081c7449be8819974c5363fa32: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:55.653016 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server e5c03261d31c4d9a91077a17e1cd0336
I20240501 07:59:55.721757 13508 rebalancer_tool.cc:1074] tablet 25cee7e51c7c438aa721d71fb82dc71e: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '53d1d2c0767b4f6195b69758a3dd6cf9' move scheduled
I20240501 07:59:55.946558 13508 rebalancer_tool.cc:1074] tablet 18b146cc4e164e1d9f9f827217c9b3ab: '655ebda1fb2c49c4af7087a760529835' -> 'e5c03261d31c4d9a91077a17e1cd0336' move scheduled
I20240501 07:59:56.077880 13508 rebalancer_tool.cc:1074] tablet 64613ea537c842f29d810e829278394e: '22ec12774a4047a8b859743dc73f8ff6' -> '60e8857abe7c41038af46ebb5a095a90' move scheduled
I20240501 07:59:56.181557 13508 rebalancer_tool.cc:1074] tablet 90f11b5693d7456d8a604e6c8726e69a: '603919bb02364c04b6c530245b8b02f2' -> 'ca177138f2c44d2eafa1646b795f7608' move scheduled
I20240501 07:59:56.225481 13508 rebalancer_tool.cc:1084] tablet f80db2e37bbc4716ad3a5200c61ee97b: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '53d1d2c0767b4f6195b69758a3dd6cf9' move ignored: Illegal state: tablet f80db2e37bbc4716ad3a5200c61ee97b: replica 53d1d2c0767b4f6195b69758a3dd6cf9 is already present
I20240501 07:59:56.225593 13508 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 07:59:56.504586 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 69f109081c7449be8819974c5363fa32: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:56.504796 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 701e50fb54354219a26221f76a7352d2: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:56.504835 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 98fb02bdb5f54b1699749e3c8d588a5e: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:56.504890 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af21d4c4fcb49a7ac0756a5ded5d57d: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:56.504925 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 384b2b245c6b4931a4fc02c6b5d647b2: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:56.504959 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6447707eb34f45d7a8d3704cd09d7786: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:56.505003 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:56.505034 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 36e20074d8dc4402bd7d9b2039213af5: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:56.505064 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 701e50fb54354219a26221f76a7352d2: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:56.505122 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af21d4c4fcb49a7ac0756a5ded5d57d: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:56.505189 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df5f9864bbf8499dbaa392bcb85701c0: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:56.505223 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 36e20074d8dc4402bd7d9b2039213af5: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:56.505261 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24b4f6e8cc8c44f6961591cac7beacb4: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:56.505307 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 69f109081c7449be8819974c5363fa32: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:56.505338 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 98fb02bdb5f54b1699749e3c8d588a5e: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:56.505402 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:56.505437 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 701e50fb54354219a26221f76a7352d2: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:56.505470 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af21d4c4fcb49a7ac0756a5ded5d57d: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:56.505504 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 384b2b245c6b4931a4fc02c6b5d647b2: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:56.505538 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6447707eb34f45d7a8d3704cd09d7786: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 53d1d2c0767b4f6195b69758a3dd6cf9
I20240501 07:59:56.789819 13508 rebalancer_tool.cc:1074] tablet ea97f34590684bcf998228245dbcd9b0: '22ec12774a4047a8b859743dc73f8ff6' -> '53d1d2c0767b4f6195b69758a3dd6cf9' move scheduled
I20240501 07:59:56.955157 13508 rebalancer_tool.cc:1074] tablet a1ebf29682024645978b3caa2d4988f3: 'e2dafc1866314669ad0de8fb7bf7fd05' -> 'e5c03261d31c4d9a91077a17e1cd0336' move scheduled
I20240501 07:59:57.031600 13508 rebalancer_tool.cc:1074] tablet c95c4f3ef08f44d1b69fb7b98426f5af: '655ebda1fb2c49c4af7087a760529835' -> '60e8857abe7c41038af46ebb5a095a90' move scheduled
I20240501 07:59:57.240449 13508 rebalancer_tool.cc:1074] tablet 53193305f868433c852755e8a8a81766: '603919bb02364c04b6c530245b8b02f2' -> 'ca177138f2c44d2eafa1646b795f7608' move scheduled
I20240501 07:59:57.445084 13508 rebalancer_tool.cc:1084] tablet 7c23e3bd7ec34876956046e5d32e4d82: '22ec12774a4047a8b859743dc73f8ff6' -> '53d1d2c0767b4f6195b69758a3dd6cf9' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 07:59:57.445188 13508 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 07:59:57.958442 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:57.959111 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6447707eb34f45d7a8d3704cd09d7786: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:57.959192 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24b4f6e8cc8c44f6961591cac7beacb4: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:57.959220 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 384b2b245c6b4931a4fc02c6b5d647b2: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:57.959247 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af21d4c4fcb49a7ac0756a5ded5d57d: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:57.959272 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df5f9864bbf8499dbaa392bcb85701c0: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:57.959295 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:57.959322 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 701e50fb54354219a26221f76a7352d2: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:57.959367 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 36e20074d8dc4402bd7d9b2039213af5: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:57.959399 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 98fb02bdb5f54b1699749e3c8d588a5e: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:57.959425 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6447707eb34f45d7a8d3704cd09d7786: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:57.959450 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24b4f6e8cc8c44f6961591cac7beacb4: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:57.959475 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 384b2b245c6b4931a4fc02c6b5d647b2: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 60e8857abe7c41038af46ebb5a095a90
I20240501 07:59:58.053807 13508 rebalancer_tool.cc:1074] tablet b9431cf5853c4148ac38855b1c083d4f: '655ebda1fb2c49c4af7087a760529835' -> '' move scheduled
I20240501 07:59:58.096626 13508 rebalancer_tool.cc:1074] tablet 867ba08a15b44d4c9e6fa92b0d518d1d: '22ec12774a4047a8b859743dc73f8ff6' -> '' move scheduled
I20240501 07:59:58.115312 13508 rebalancer_tool.cc:1084] tablet b21b6e2023e842739a7c0cc946e7c2fd: '655ebda1fb2c49c4af7087a760529835' -> '60e8857abe7c41038af46ebb5a095a90' move ignored: Illegal state: tablet b21b6e2023e842739a7c0cc946e7c2fd: replica 60e8857abe7c41038af46ebb5a095a90 is already present
I20240501 07:59:58.115423 13508 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 07:59:58.551822 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24b4f6e8cc8c44f6961591cac7beacb4: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:58.551944 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6447707eb34f45d7a8d3704cd09d7786: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 60e8857abe7c41038af46ebb5a095a90
W20240501 07:59:58.552011 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 384b2b245c6b4931a4fc02c6b5d647b2: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:58.552073 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1deda0b4866b43c9ba7c69dae5d65e4a: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:58.552143 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af21d4c4fcb49a7ac0756a5ded5d57d: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server e5c03261d31c4d9a91077a17e1cd0336
W20240501 07:59:58.552187 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 24b4f6e8cc8c44f6961591cac7beacb4: could not find any suitable replica to move from server e2dafc1866314669ad0de8fb7bf7fd05 to server ca177138f2c44d2eafa1646b795f7608
W20240501 07:59:58.552227 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 36e20074d8dc4402bd7d9b2039213af5: could not find any suitable replica to move from server 655ebda1fb2c49c4af7087a760529835 to server 53d1d2c0767b4f6195b69758a3dd6cf9
W20240501 07:59:58.552268 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6447707eb34f45d7a8d3704cd09d7786: could not find any suitable replica to move from server 603919bb02364c04b6c530245b8b02f2 to server 60e8857abe7c41038af46ebb5a095a90
I20240501 07:59:58.602577 13508 rebalancer_tool.cc:1074] tablet f80db2e37bbc4716ad3a5200c61ee97b: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '' move scheduled
I20240501 07:59:58.618017 13508 rebalancer_tool.cc:1074] tablet f2e63d24d7df4fa3b2b26695e70ebe9f: '603919bb02364c04b6c530245b8b02f2' -> '' move scheduled
I20240501 07:59:58.631806 13508 rebalancer_tool.cc:1074] tablet 3f6e4ae97e884219ac7d0d94997d9cc9: '22ec12774a4047a8b859743dc73f8ff6' -> '' move scheduled
I20240501 08:00:03.024878 13508 rebalancer_tool.cc:1136] tablet 18b146cc4e164e1d9f9f827217c9b3ab: '655ebda1fb2c49c4af7087a760529835' -> 'e5c03261d31c4d9a91077a17e1cd0336' move completed: OK
I20240501 08:00:03.946084 13508 rebalancer_tool.cc:1136] tablet 64613ea537c842f29d810e829278394e: '22ec12774a4047a8b859743dc73f8ff6' -> '60e8857abe7c41038af46ebb5a095a90' move completed: OK
I20240501 08:00:05.189677 13508 rebalancer_tool.cc:1123] tablet f2e63d24d7df4fa3b2b26695e70ebe9f: '603919bb02364c04b6c530245b8b02f2' -> '' move is abandoned: Illegal state: Replica e2dafc1866314669ad0de8fb7bf7fd05 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } }
I20240501 08:00:05.222719 13508 rebalancer_tool.cc:1136] tablet 90f11b5693d7456d8a604e6c8726e69a: '603919bb02364c04b6c530245b8b02f2' -> 'ca177138f2c44d2eafa1646b795f7608' move completed: OK
I20240501 08:00:05.234189 13508 rebalancer_tool.cc:1136] tablet a1ebf29682024645978b3caa2d4988f3: 'e2dafc1866314669ad0de8fb7bf7fd05' -> 'e5c03261d31c4d9a91077a17e1cd0336' move completed: OK
I20240501 08:00:05.319425 13508 rebalancer_tool.cc:1136] tablet 3f6e4ae97e884219ac7d0d94997d9cc9: '22ec12774a4047a8b859743dc73f8ff6' -> '' move completed: OK
I20240501 08:00:05.410681 13508 rebalancer_tool.cc:1136] tablet f80db2e37bbc4716ad3a5200c61ee97b: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '' move completed: OK
I20240501 08:00:05.420076 13508 rebalancer_tool.cc:1136] tablet 867ba08a15b44d4c9e6fa92b0d518d1d: '22ec12774a4047a8b859743dc73f8ff6' -> '' move completed: OK
I20240501 08:00:06.708261 13508 rebalancer_tool.cc:1136] tablet c95c4f3ef08f44d1b69fb7b98426f5af: '655ebda1fb2c49c4af7087a760529835' -> '60e8857abe7c41038af46ebb5a095a90' move completed: OK
I20240501 08:00:07.273061 13508 rebalancer_tool.cc:1136] tablet ea97f34590684bcf998228245dbcd9b0: '22ec12774a4047a8b859743dc73f8ff6' -> '53d1d2c0767b4f6195b69758a3dd6cf9' move completed: OK
I20240501 08:00:07.286973 13508 rebalancer_tool.cc:1074] tablet 164fd265561e47128e7ff585b8cfab07: '22ec12774a4047a8b859743dc73f8ff6' -> '' move scheduled
I20240501 08:00:07.365630 13508 rebalancer_tool.cc:1074] tablet 5b7de08d2f004cd8abc2bbe1cf7b1444: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '' move scheduled
I20240501 08:00:07.376718 13508 rebalancer_tool.cc:1074] tablet 48b0ff0769dc426190f038811155c741: '655ebda1fb2c49c4af7087a760529835' -> '' move scheduled
I20240501 08:00:07.403484 13508 rebalancer_tool.cc:1136] tablet 48b0ff0769dc426190f038811155c741: '655ebda1fb2c49c4af7087a760529835' -> '' move completed: OK
I20240501 08:00:07.447868 13508 rebalancer_tool.cc:1136] tablet 5b7de08d2f004cd8abc2bbe1cf7b1444: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '' move completed: OK
I20240501 08:00:07.531679 13508 rebalancer_tool.cc:1136] tablet 164fd265561e47128e7ff585b8cfab07: '22ec12774a4047a8b859743dc73f8ff6' -> '' move completed: OK
I20240501 08:00:07.554962 13508 rebalancer_tool.cc:1136] tablet b9431cf5853c4148ac38855b1c083d4f: '655ebda1fb2c49c4af7087a760529835' -> '' move completed: OK
I20240501 08:00:07.581941 13508 rebalancer_tool.cc:1136] tablet 53193305f868433c852755e8a8a81766: '603919bb02364c04b6c530245b8b02f2' -> 'ca177138f2c44d2eafa1646b795f7608' move completed: OK
I20240501 08:00:07.637400 13508 rebalancer_tool.cc:1136] tablet 25cee7e51c7c438aa721d71fb82dc71e: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '53d1d2c0767b4f6195b69758a3dd6cf9' move completed: OK
I20240501 08:00:07.649916 13508 rebalancer_tool.cc:1074] tablet f61e70ba02384f81b9ae36b1c65465b9: '655ebda1fb2c49c4af7087a760529835' -> '' move scheduled
I20240501 08:00:07.727699 13508 rebalancer_tool.cc:1074] tablet ed88ea311f474ab28cc6fbe6069258bf: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '' move scheduled
I20240501 08:00:07.826473 13508 rebalancer_tool.cc:1074] tablet 6d214be86534458091f54a386d7d3334: '603919bb02364c04b6c530245b8b02f2' -> '' move scheduled
I20240501 08:00:07.865075 13508 rebalancer_tool.cc:1074] tablet 93602521de174997bda6702d25aafce1: '655ebda1fb2c49c4af7087a760529835' -> '' move scheduled
I20240501 08:00:07.888860 13508 rebalancer_tool.cc:1136] tablet 93602521de174997bda6702d25aafce1: '655ebda1fb2c49c4af7087a760529835' -> '' move completed: OK
I20240501 08:00:07.983952 13508 rebalancer_tool.cc:1136] tablet 6d214be86534458091f54a386d7d3334: '603919bb02364c04b6c530245b8b02f2' -> '' move completed: OK
I20240501 08:00:08.000902 13508 rebalancer_tool.cc:1136] tablet ed88ea311f474ab28cc6fbe6069258bf: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '' move completed: OK
I20240501 08:00:08.010702 13508 rebalancer_tool.cc:1136] tablet f61e70ba02384f81b9ae36b1c65465b9: '655ebda1fb2c49c4af7087a760529835' -> '' move completed: OK
I20240501 08:00:08.021411 13508 rebalancer_tool.cc:1074] tablet b4d15e19eed54aefb6b96276fad98324: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '' move scheduled
I20240501 08:00:08.052275 13508 rebalancer_tool.cc:1074] tablet bd8348990f1b496bb62d8c0484193442: '603919bb02364c04b6c530245b8b02f2' -> '' move scheduled
I20240501 08:00:08.099795 13508 rebalancer_tool.cc:1136] tablet bd8348990f1b496bb62d8c0484193442: '603919bb02364c04b6c530245b8b02f2' -> '' move completed: OK
I20240501 08:00:08.197058 13508 rebalancer_tool.cc:1136] tablet b4d15e19eed54aefb6b96276fad98324: 'e2dafc1866314669ad0de8fb7bf7fd05' -> '' move completed: OK
W20240501 08:00:08.675855 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 69f109081c7449be8819974c5363fa32: could not find any suitable replica to move from server 60e8857abe7c41038af46ebb5a095a90 to server e2dafc1866314669ad0de8fb7bf7fd05
W20240501 08:00:08.676072 13508 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 36e20074d8dc4402bd7d9b2039213af5: could not find any suitable replica to move from server 22ec12774a4047a8b859743dc73f8ff6 to server 603919bb02364c04b6c530245b8b02f2
I20240501 08:00:08.931052 13508 rebalancer_tool.cc:1074] tablet ea97f34590684bcf998228245dbcd9b0: '655ebda1fb2c49c4af7087a760529835' -> 'e5c03261d31c4d9a91077a17e1cd0336' move scheduled
I20240501 08:00:09.057595 13508 rebalancer_tool.cc:1074] tablet ed88ea311f474ab28cc6fbe6069258bf: '22ec12774a4047a8b859743dc73f8ff6' -> 'e2dafc1866314669ad0de8fb7bf7fd05' move scheduled
I20240501 08:00:09.165762 13508 rebalancer_tool.cc:1074] tablet c95c4f3ef08f44d1b69fb7b98426f5af: '60e8857abe7c41038af46ebb5a095a90' -> 'ca177138f2c44d2eafa1646b795f7608' move scheduled
I20240501 08:00:09.498454 13508 rebalancer_tool.cc:1074] tablet 17bb1ff829bc47a6a0bd6030517652aa: '53d1d2c0767b4f6195b69758a3dd6cf9' -> '60e8857abe7c41038af46ebb5a095a90' move scheduled
I20240501 08:00:09.568583 13508 rebalancer_tool.cc:1074] tablet 3f6e4ae97e884219ac7d0d94997d9cc9: '655ebda1fb2c49c4af7087a760529835' -> '' move scheduled
I20240501 08:00:09.846859 13508 rebalancer_tool.cc:1074] tablet f80db2e37bbc4716ad3a5200c61ee97b: 'ca177138f2c44d2eafa1646b795f7608' -> 'e5c03261d31c4d9a91077a17e1cd0336' move scheduled
I20240501 08:00:10.005792 13508 rebalancer_tool.cc:1074] tablet 5b7de08d2f004cd8abc2bbe1cf7b1444: 'ca177138f2c44d2eafa1646b795f7608' -> '22ec12774a4047a8b859743dc73f8ff6' move scheduled
I20240501 08:00:10.021176 13508 rebalancer_tool.cc:1074] tablet 6d214be86534458091f54a386d7d3334: '60e8857abe7c41038af46ebb5a095a90' -> '' move scheduled
W20240501 08:00:10.516767 22437 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:00:10.309200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:42805 (local address 127.0.0.1:60344)
0501 08:00:10.309383 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0501 08:00:10.309397 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 08:00:10.309478 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:00:10.516129 (+206651us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":91}
W20240501 08:00:10.862774 22753 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:00:10.791845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.13
I20240501 08:05:10.889981 11531 consensus_queue.cc:237] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
I20240501 08:05:10.891211 11400 raft_consensus.cc:1261] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:05:10.891866 32527 consensus_queue.cc:1035] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:05:10.893556 32535 raft_consensus.cc:2932] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 08:05:10.894027 11400 raft_consensus.cc:2932] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
W20240501 08:05:10.896183 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:10.896378 11126 catalog_manager.cc:5484] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 7 to 8, VOTER ca177138f2c44d2eafa1646b795f7608 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 08:05:10.900079 11531 consensus_queue.cc:237] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:05:10.901209 11400 raft_consensus.cc:1261] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:05:10.901599 32527 consensus_queue.cc:1035] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 08:05:10.901818 11465 consensus_peers.cc:485] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403): Couldn't send request to peer 603919bb02364c04b6c530245b8b02f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4707b952d174a9bb877ad8bd7a11268. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:10.903172 32535 raft_consensus.cc:2932] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:05:10.903625 11400 raft_consensus.cc:2932] T b4707b952d174a9bb877ad8bd7a11268 P 655ebda1fb2c49c4af7087a760529835 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
I20240501 08:05:10.905164 11114 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet b4707b952d174a9bb877ad8bd7a11268 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:05:10.905926 11121 catalog_manager.cc:5484] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6 reported cstate change: config changed from index 8 to 9, VOTER 603919bb02364c04b6c530245b8b02f2 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } }
W20240501 08:05:10.907349 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:10.965124 339 ts_tablet_manager.cc:902] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: Initiating tablet copy from peer 655ebda1fb2c49c4af7087a760529835 (127.0.99.130:41575)
I20240501 08:05:10.965730 339 tablet_copy_client.cc:308] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41575
I20240501 08:05:10.966353 11420 tablet_copy_service.cc:140] P 655ebda1fb2c49c4af7087a760529835: Received BeginTabletCopySession request for tablet 7c23e3bd7ec34876956046e5d32e4d82 from peer 603919bb02364c04b6c530245b8b02f2 ({username='slave'} at 127.0.99.132:52989)
I20240501 08:05:10.966488 11420 tablet_copy_service.cc:161] P 655ebda1fb2c49c4af7087a760529835: Beginning new tablet copy session on tablet 7c23e3bd7ec34876956046e5d32e4d82 from peer 603919bb02364c04b6c530245b8b02f2 at {username='slave'} at 127.0.99.132:52989: session id = 603919bb02364c04b6c530245b8b02f2-7c23e3bd7ec34876956046e5d32e4d82
I20240501 08:05:10.967779 11420 tablet_copy_source_session.cc:205] T 7c23e3bd7ec34876956046e5d32e4d82 P 655ebda1fb2c49c4af7087a760529835: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:05:10.968715 339 ts_tablet_manager.cc:1875] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:05:10.971938 339 ts_tablet_manager.cc:1888] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 08:05:10.972718 339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c23e3bd7ec34876956046e5d32e4d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:10.974133 339 tablet_copy_client.cc:783] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: tablet copy: Starting download of 5 data blocks...
I20240501 08:05:10.976806 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:10.977880 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:10.980572 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:10.980803 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:10.985102 339 tablet_copy_client.cc:647] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:05:10.986344 339 tablet_copy_client.cc:523] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:05:10.987926 339 tablet_bootstrap.cc:492] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 08:05:10.990312 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:11.002278 339 tablet_bootstrap.cc:492] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:11.002671 339 tablet_bootstrap.cc:492] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: Bootstrap complete.
I20240501 08:05:11.002882 339 ts_tablet_manager.cc:1367] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20240501 08:05:11.003427 339 raft_consensus.cc:348] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:05:11.003535 339 raft_consensus.cc:729] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 08:05:11.003686 339 consensus_queue.cc:260] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:05:11.004098 339 ts_tablet_manager.cc:1398] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:05:11.006531 11420 tablet_copy_service.cc:342] P 655ebda1fb2c49c4af7087a760529835: Request end of tablet copy session 603919bb02364c04b6c530245b8b02f2-7c23e3bd7ec34876956046e5d32e4d82 received from {username='slave'} at 127.0.99.132:52989
I20240501 08:05:11.006628 11420 tablet_copy_service.cc:433] P 655ebda1fb2c49c4af7087a760529835: ending tablet copy session 603919bb02364c04b6c530245b8b02f2-7c23e3bd7ec34876956046e5d32e4d82 on tablet 7c23e3bd7ec34876956046e5d32e4d82 with peer 603919bb02364c04b6c530245b8b02f2
I20240501 08:05:11.010270 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling UndoDeltaBlockGCOp(b9431cf5853c4148ac38855b1c083d4f): 4836 bytes on disk
I20240501 08:05:11.010777 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: UndoDeltaBlockGCOp(b9431cf5853c4148ac38855b1c083d4f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240501 08:05:11.028586 11661 raft_consensus.cc:1203] T b21b6e2023e842739a7c0cc946e7c2fd P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 08:05:11.071125 339 ts_tablet_manager.cc:902] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: Initiating tablet copy from peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 08:05:11.071643 339 tablet_copy_client.cc:308] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37957
I20240501 08:05:11.072158 11551 tablet_copy_service.cc:140] P 22ec12774a4047a8b859743dc73f8ff6: Received BeginTabletCopySession request for tablet 48b0ff0769dc426190f038811155c741 from peer 603919bb02364c04b6c530245b8b02f2 ({username='slave'} at 127.0.99.132:54785)
I20240501 08:05:11.072269 11551 tablet_copy_service.cc:161] P 22ec12774a4047a8b859743dc73f8ff6: Beginning new tablet copy session on tablet 48b0ff0769dc426190f038811155c741 from peer 603919bb02364c04b6c530245b8b02f2 at {username='slave'} at 127.0.99.132:54785: session id = 603919bb02364c04b6c530245b8b02f2-48b0ff0769dc426190f038811155c741
W20240501 08:05:11.073381 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:11.073748 11551 tablet_copy_source_session.cc:205] T 48b0ff0769dc426190f038811155c741 P 22ec12774a4047a8b859743dc73f8ff6: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:05:11.074662 339 ts_tablet_manager.cc:1875] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 08:05:11.076790 339 ts_tablet_manager.cc:1888] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 08:05:11.076979 339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b0ff0769dc426190f038811155c741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:11.077984 339 tablet_copy_client.cc:783] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: tablet copy: Starting download of 5 data blocks...
I20240501 08:05:11.082459 32211 raft_consensus.cc:998] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2: Attempting to remove follower e5c03261d31c4d9a91077a17e1cd0336 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5c03261d31c4d9a91077a17e1cd0336 for more than 300 seconds (300.409s)
I20240501 08:05:11.082804 32211 consensus_queue.cc:237] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:05:11.084017 13185 raft_consensus.cc:1261] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:05:11.085706 546 consensus_queue.cc:1035] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:05:11.088429 339 tablet_copy_client.cc:647] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:05:11.089936 339 tablet_copy_client.cc:523] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:05:11.091446 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:11.091588 339 tablet_bootstrap.cc:492] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 08:05:11.095432 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:11.101521 339 tablet_bootstrap.cc:492] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:11.101817 339 tablet_bootstrap.cc:492] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: Bootstrap complete.
I20240501 08:05:11.102003 339 ts_tablet_manager.cc:1367] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240501 08:05:11.102605 339 raft_consensus.cc:348] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:05:11.102730 339 raft_consensus.cc:729] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 08:05:11.102903 339 consensus_queue.cc:260] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:05:11.103176 339 ts_tablet_manager.cc:1398] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 08:05:11.103262 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:11.104106 32211 raft_consensus.cc:2932] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:11.104609 11551 tablet_copy_service.cc:342] P 22ec12774a4047a8b859743dc73f8ff6: Request end of tablet copy session 603919bb02364c04b6c530245b8b02f2-48b0ff0769dc426190f038811155c741 received from {username='slave'} at 127.0.99.132:54785
I20240501 08:05:11.104678 11551 tablet_copy_service.cc:433] P 22ec12774a4047a8b859743dc73f8ff6: ending tablet copy session 603919bb02364c04b6c530245b8b02f2-48b0ff0769dc426190f038811155c741 on tablet 48b0ff0769dc426190f038811155c741 with peer 603919bb02364c04b6c530245b8b02f2
I20240501 08:05:11.106289 13185 raft_consensus.cc:2932] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:11.109120 11121 catalog_manager.cc:5484] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9 reported cstate change: config changed from index 7 to 8, VOTER e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } }
I20240501 08:05:11.113638 11661 consensus_queue.cc:237] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
W20240501 08:05:11.115478 11597 consensus_peers.cc:485] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 -> Peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085): Couldn't send request to peer e2dafc1866314669ad0de8fb7bf7fd05. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f61e70ba02384f81b9ae36b1c65465b9. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:11.116649 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:11.117136 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:11.123219 13185 raft_consensus.cc:1261] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Refusing update from remote peer 603919bb02364c04b6c530245b8b02f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:05:11.127365 546 consensus_queue.cc:1035] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:05:11.131268 556 raft_consensus.cc:2932] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:05:11.133225 11112 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f61e70ba02384f81b9ae36b1c65465b9 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 08:05:11.133764 13185 raft_consensus.cc:2932] T f61e70ba02384f81b9ae36b1c65465b9 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:05:11.135836 11126 catalog_manager.cc:5484] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2 reported cstate change: config changed from index 8 to 9, VOTER e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "603919bb02364c04b6c530245b8b02f2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } }
I20240501 08:05:11.163867 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling UndoDeltaBlockGCOp(48b0ff0769dc426190f038811155c741): 4836 bytes on disk
I20240501 08:05:11.164378 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: UndoDeltaBlockGCOp(48b0ff0769dc426190f038811155c741) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240501 08:05:11.166016 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling UndoDeltaBlockGCOp(7c23e3bd7ec34876956046e5d32e4d82): 4586 bytes on disk
I20240501 08:05:11.166522 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: UndoDeltaBlockGCOp(7c23e3bd7ec34876956046e5d32e4d82) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240501 08:05:11.198678 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:11.198827 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:11.202625 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:11.210813 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:11.212534 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:11.250451 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 08:05:11.269160 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:11.272779 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:11.273051 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:11.273887 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:11.291440 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:11.341730 13185 raft_consensus.cc:1203] T b9431cf5853c4148ac38855b1c083d4f P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240501 08:05:11.410318 339 ts_tablet_manager.cc:902] T b4707b952d174a9bb877ad8bd7a11268 P 603919bb02364c04b6c530245b8b02f2: Initiating tablet copy from peer 22ec12774a4047a8b859743dc73f8ff6 (127.0.99.131:37957)
I20240501 08:05:11.410856 339 tablet_copy_client.cc:308] T b4707b952d174a9bb877ad8bd7a11268 P 603919bb02364c04b6c530245b8b02f2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:37957
I20240501 08:05:11.411674 11551 tablet_copy_service.cc:140] P 22ec12774a4047a8b859743dc73f8ff6: Received BeginTabletCopySession request for tablet b4707b952d174a9bb877ad8bd7a11268 from peer 603919bb02364c04b6c530245b8b02f2 ({username='slave'} at 127.0.99.132:54785)
I20240501 08:05:11.411798 11551 tablet_copy_service.cc:161] P 22ec12774a4047a8b859743dc73f8ff6: Beginning new tablet copy session on tablet b4707b952d174a9bb877ad8bd7a11268 from peer 603919bb02364c04b6c530245b8b02f2 at {username='slave'} at 127.0.99.132:54785: session id = 603919bb02364c04b6c530245b8b02f2-b4707b952d174a9bb877ad8bd7a11268
I20240501 08:05:11.412506 678 ts_tablet_manager.cc:902] T 18b146cc4e164e1d9f9f827217c9b3ab P 53d1d2c0767b4f6195b69758a3dd6cf9: Initiating tablet copy from peer e2dafc1866314669ad0de8fb7bf7fd05 (127.0.99.129:46085)
I20240501 08:05:11.412943 678 tablet_copy_client.cc:308] T 18b146cc4e164e1d9f9f827217c9b3ab P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46085
I20240501 08:05:11.413040 11551 tablet_copy_source_session.cc:205] T b4707b952d174a9bb877ad8bd7a11268 P 22ec12774a4047a8b859743dc73f8ff6: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:05:11.414309 339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4707b952d174a9bb877ad8bd7a11268. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:11.414361 11290 tablet_copy_service.cc:140] P e2dafc1866314669ad0de8fb7bf7fd05: Received BeginTabletCopySession request for tablet 18b146cc4e164e1d9f9f827217c9b3ab from peer 53d1d2c0767b4f6195b69758a3dd6cf9 ({username='slave'} at 127.0.99.134:37773)
I20240501 08:05:11.414659 11290 tablet_copy_service.cc:161] P e2dafc1866314669ad0de8fb7bf7fd05: Beginning new tablet copy session on tablet 18b146cc4e164e1d9f9f827217c9b3ab from peer 53d1d2c0767b4f6195b69758a3dd6cf9 at {username='slave'} at 127.0.99.134:37773: session id = 53d1d2c0767b4f6195b69758a3dd6cf9-18b146cc4e164e1d9f9f827217c9b3ab
I20240501 08:05:11.415752 11290 tablet_copy_source_session.cc:205] T 18b146cc4e164e1d9f9f827217c9b3ab P e2dafc1866314669ad0de8fb7bf7fd05: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:05:11.416743 678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b146cc4e164e1d9f9f827217c9b3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:11.417737 11661 raft_consensus.cc:1203] T 7c23e3bd7ec34876956046e5d32e4d82 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 08:05:11.417730 339 tablet_copy_client.cc:783] T b4707b952d174a9bb877ad8bd7a11268 P 603919bb02364c04b6c530245b8b02f2: tablet copy: Starting download of 5 data blocks...
I20240501 08:05:11.421211 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:11.425338 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:11.428088 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:11.428267 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:11.435606 339 tablet_copy_client.cc:647] T b4707b952d174a9bb877ad8bd7a11268 P 603919bb02364c04b6c530245b8b02f2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:05:11.438261 678 tablet_copy_client.cc:783] T 18b146cc4e164e1d9f9f827217c9b3ab P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 5 data blocks...
I20240501 08:05:11.439285 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:11.448973 339 tablet_copy_client.cc:523] T b4707b952d174a9bb877ad8bd7a11268 P 603919bb02364c04b6c530245b8b02f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:05:11.450649 339 tablet_bootstrap.cc:492] T b4707b952d174a9bb877ad8bd7a11268 P 603919bb02364c04b6c530245b8b02f2: Bootstrap starting.
I20240501 08:05:11.451349 32527 raft_consensus.cc:998] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6: Attempting to remove follower 60e8857abe7c41038af46ebb5a095a90 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 60e8857abe7c41038af46ebb5a095a90 for more than 300 seconds (300.064s)
I20240501 08:05:11.451757 32527 consensus_queue.cc:237] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } }
W20240501 08:05:11.455098 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:05:11.455713 11400 raft_consensus.cc:1261] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 08:05:11.456053 11661 raft_consensus.cc:1261] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Refusing update from remote peer 22ec12774a4047a8b859743dc73f8ff6: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 08:05:11.456485 32527 consensus_queue.cc:1035] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:05:11.456773 684 consensus_queue.cc:1035] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 08:05:11.462627 678 tablet_copy_client.cc:647] T 18b146cc4e164e1d9f9f827217c9b3ab P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Starting download of 1 WAL segments...
I20240501 08:05:11.464078 678 tablet_copy_client.cc:523] T 18b146cc4e164e1d9f9f827217c9b3ab P 53d1d2c0767b4f6195b69758a3dd6cf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:05:11.465638 678 tablet_bootstrap.cc:492] T 18b146cc4e164e1d9f9f827217c9b3ab P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap starting.
I20240501 08:05:11.467825 686 raft_consensus.cc:2932] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 08:05:11.470757 11400 raft_consensus.cc:2932] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 08:05:11.474052 11119 catalog_manager.cc:5484] T 5e8550ea95984d08937b9b3e2af12bec P 655ebda1fb2c49c4af7087a760529835 reported cstate change: config changed from index 10 to 11, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New cstate: current_term: 2 leader_uuid: "22ec12774a4047a8b859743dc73f8ff6" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 08:05:11.475297 11661 raft_consensus.cc:2932] T 5e8550ea95984d08937b9b3e2af12bec P 603919bb02364c04b6c530245b8b02f2 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } }
I20240501 08:05:11.478960 678 tablet_bootstrap.cc:492] T 18b146cc4e164e1d9f9f827217c9b3ab P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:11.479379 678 tablet_bootstrap.cc:492] T 18b146cc4e164e1d9f9f827217c9b3ab P 53d1d2c0767b4f6195b69758a3dd6cf9: Bootstrap complete.
I20240501 08:05:11.479585 678 ts_tablet_manager.cc:1367] T 18b146cc4e164e1d9f9f827217c9b3ab P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20240501 08:05:11.480139 339 tablet_bootstrap.cc:492] T b4707b952d174a9bb877ad8bd7a11268 P 603919bb02364c04b6c530245b8b02f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:11.480100 678 raft_consensus.cc:348] T 18b146cc4e164e1d9f9f827217c9b3ab P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:05:11.480214 678 raft_consensus.cc:729] T 18b146cc4e164e1d9f9f827217c9b3ab P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d1d2c0767b4f6195b69758a3dd6cf9, State: Initialized, Role: FOLLOWER
I20240501 08:05:11.480342 678 consensus_queue.cc:260] T 18b146cc4e164e1d9f9f827217c9b3ab P 53d1d2c0767b4f6195b69758a3dd6cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } } peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } }
I20240501 08:05:11.480495 339 tablet_bootstrap.cc:492] T b4707b952d174a9bb877ad8bd7a11268 P 603919bb02364c04b6c530245b8b02f2: Bootstrap complete.
I20240501 08:05:11.480646 678 ts_tablet_manager.cc:1398] T 18b146cc4e164e1d9f9f827217c9b3ab P 53d1d2c0767b4f6195b69758a3dd6cf9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 08:05:11.480667 339 ts_tablet_manager.cc:1367] T b4707b952d174a9bb877ad8bd7a11268 P 603919bb02364c04b6c530245b8b02f2: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.000s
I20240501 08:05:11.481142 11290 tablet_copy_service.cc:342] P e2dafc1866314669ad0de8fb7bf7fd05: Request end of tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-18b146cc4e164e1d9f9f827217c9b3ab received from {username='slave'} at 127.0.99.134:37773
I20240501 08:05:11.481220 11290 tablet_copy_service.cc:433] P e2dafc1866314669ad0de8fb7bf7fd05: ending tablet copy session 53d1d2c0767b4f6195b69758a3dd6cf9-18b146cc4e164e1d9f9f827217c9b3ab on tablet 18b146cc4e164e1d9f9f827217c9b3ab with peer 53d1d2c0767b4f6195b69758a3dd6cf9
I20240501 08:05:11.481437 339 raft_consensus.cc:348] T b4707b952d174a9bb877ad8bd7a11268 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:05:11.482472 339 raft_consensus.cc:729] T b4707b952d174a9bb877ad8bd7a11268 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603919bb02364c04b6c530245b8b02f2, State: Initialized, Role: FOLLOWER
I20240501 08:05:11.482621 339 consensus_queue.cc:260] T b4707b952d174a9bb877ad8bd7a11268 P 603919bb02364c04b6c530245b8b02f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22ec12774a4047a8b859743dc73f8ff6" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 37957 } } peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } }
I20240501 08:05:11.483001 339 ts_tablet_manager.cc:1398] T b4707b952d174a9bb877ad8bd7a11268 P 603919bb02364c04b6c530245b8b02f2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:05:11.483451 11551 tablet_copy_service.cc:342] P 22ec12774a4047a8b859743dc73f8ff6: Request end of tablet copy session 603919bb02364c04b6c530245b8b02f2-b4707b952d174a9bb877ad8bd7a11268 received from {username='slave'} at 127.0.99.132:54785
I20240501 08:05:11.483527 11551 tablet_copy_service.cc:433] P 22ec12774a4047a8b859743dc73f8ff6: ending tablet copy session 603919bb02364c04b6c530245b8b02f2-b4707b952d174a9bb877ad8bd7a11268 on tablet b4707b952d174a9bb877ad8bd7a11268 with peer 603919bb02364c04b6c530245b8b02f2
I20240501 08:05:11.513087 13233 maintenance_manager.cc:419] P 53d1d2c0767b4f6195b69758a3dd6cf9: Scheduling UndoDeltaBlockGCOp(18b146cc4e164e1d9f9f827217c9b3ab): 4840 bytes on disk
I20240501 08:05:11.513576 13123 maintenance_manager.cc:643] P 53d1d2c0767b4f6195b69758a3dd6cf9: UndoDeltaBlockGCOp(18b146cc4e164e1d9f9f827217c9b3ab) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240501 08:05:11.519081 726 ts_tablet_manager.cc:902] T f61e70ba02384f81b9ae36b1c65465b9 P e2dafc1866314669ad0de8fb7bf7fd05: Initiating tablet copy from peer 603919bb02364c04b6c530245b8b02f2 (127.0.99.132:44403)
I20240501 08:05:11.519589 726 tablet_copy_client.cc:308] T f61e70ba02384f81b9ae36b1c65465b9 P e2dafc1866314669ad0de8fb7bf7fd05: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:44403
I20240501 08:05:11.521054 11683 tablet_copy_service.cc:140] P 603919bb02364c04b6c530245b8b02f2: Received BeginTabletCopySession request for tablet f61e70ba02384f81b9ae36b1c65465b9 from peer e2dafc1866314669ad0de8fb7bf7fd05 ({username='slave'} at 127.0.99.129:33047)
I20240501 08:05:11.521173 11683 tablet_copy_service.cc:161] P 603919bb02364c04b6c530245b8b02f2: Beginning new tablet copy session on tablet f61e70ba02384f81b9ae36b1c65465b9 from peer e2dafc1866314669ad0de8fb7bf7fd05 at {username='slave'} at 127.0.99.129:33047: session id = e2dafc1866314669ad0de8fb7bf7fd05-f61e70ba02384f81b9ae36b1c65465b9
I20240501 08:05:11.522311 11683 tablet_copy_source_session.cc:205] T f61e70ba02384f81b9ae36b1c65465b9 P 603919bb02364c04b6c530245b8b02f2: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:05:11.523531 726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61e70ba02384f81b9ae36b1c65465b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:05:11.526101 726 tablet_copy_client.cc:783] T f61e70ba02384f81b9ae36b1c65465b9 P e2dafc1866314669ad0de8fb7bf7fd05: tablet copy: Starting download of 5 data blocks...
W20240501 08:05:11.526849 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:11.532593 726 tablet_copy_client.cc:647] T f61e70ba02384f81b9ae36b1c65465b9 P e2dafc1866314669ad0de8fb7bf7fd05: tablet copy: Starting download of 1 WAL segments...
I20240501 08:05:11.534003 726 tablet_copy_client.cc:523] T f61e70ba02384f81b9ae36b1c65465b9 P e2dafc1866314669ad0de8fb7bf7fd05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:05:11.535809 726 tablet_bootstrap.cc:492] T f61e70ba02384f81b9ae36b1c65465b9 P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap starting.
I20240501 08:05:11.549257 726 tablet_bootstrap.cc:492] T f61e70ba02384f81b9ae36b1c65465b9 P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:05:11.549705 726 tablet_bootstrap.cc:492] T f61e70ba02384f81b9ae36b1c65465b9 P e2dafc1866314669ad0de8fb7bf7fd05: Bootstrap complete.
I20240501 08:05:11.549942 726 ts_tablet_manager.cc:1367] T f61e70ba02384f81b9ae36b1c65465b9 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20240501 08:05:11.550493 726 raft_consensus.cc:348] T f61e70ba02384f81b9ae36b1c65465b9 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 08:05:11.550621 726 raft_consensus.cc:729] T f61e70ba02384f81b9ae36b1c65465b9 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dafc1866314669ad0de8fb7bf7fd05, State: Initialized, Role: FOLLOWER
I20240501 08:05:11.550755 726 consensus_queue.cc:260] T f61e70ba02384f81b9ae36b1c65465b9 P e2dafc1866314669ad0de8fb7bf7fd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "603919bb02364c04b6c530245b8b02f2" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 44403 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "e2dafc1866314669ad0de8fb7bf7fd05" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46085 } }
I20240501 08:05:11.554119 726 ts_tablet_manager.cc:1398] T f61e70ba02384f81b9ae36b1c65465b9 P e2dafc1866314669ad0de8fb7bf7fd05: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:05:11.555178 11683 tablet_copy_service.cc:342] P 603919bb02364c04b6c530245b8b02f2: Request end of tablet copy session e2dafc1866314669ad0de8fb7bf7fd05-f61e70ba02384f81b9ae36b1c65465b9 received from {username='slave'} at 127.0.99.129:33047
I20240501 08:05:11.555276 11683 tablet_copy_service.cc:433] P 603919bb02364c04b6c530245b8b02f2: ending tablet copy session e2dafc1866314669ad0de8fb7bf7fd05-f61e70ba02384f81b9ae36b1c65465b9 on tablet f61e70ba02384f81b9ae36b1c65465b9 with peer e2dafc1866314669ad0de8fb7bf7fd05
I20240501 08:05:11.567977 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:11.568203 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:11.571426 11661 raft_consensus.cc:1203] T 48b0ff0769dc426190f038811155c741 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 08:05:11.572950 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:11.580686 11317 maintenance_manager.cc:419] P e2dafc1866314669ad0de8fb7bf7fd05: Scheduling UndoDeltaBlockGCOp(f61e70ba02384f81b9ae36b1c65465b9): 4586 bytes on disk
I20240501 08:05:11.581312 11207 maintenance_manager.cc:643] P e2dafc1866314669ad0de8fb7bf7fd05: UndoDeltaBlockGCOp(f61e70ba02384f81b9ae36b1c65465b9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240501 08:05:11.583985 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:11.584451 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:11.605206 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:11.610342 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:11.628046 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:11.631006 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:11.646823 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:11.656697 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 37 similar messages]
I20240501 08:05:11.669073 11710 maintenance_manager.cc:419] P 603919bb02364c04b6c530245b8b02f2: Scheduling UndoDeltaBlockGCOp(b4707b952d174a9bb877ad8bd7a11268): 4586 bytes on disk
I20240501 08:05:11.669648 11600 maintenance_manager.cc:643] P 603919bb02364c04b6c530245b8b02f2: UndoDeltaBlockGCOp(b4707b952d174a9bb877ad8bd7a11268) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240501 08:05:11.672065 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 08:05:11.687258 5411 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:05:11.687440 5411 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:05:11.688025 5411 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
W20240501 08:05:11.688974 13122 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:11.689163 13121 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:11.689224 13121 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9; no voters: 60e8857abe7c41038af46ebb5a095a90, ca177138f2c44d2eafa1646b795f7608
I20240501 08:05:11.689375 5411 raft_consensus.cc:2726] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:05:11.726544 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:11.726969 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:11.730686 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:11.731204 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:11.744498 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:11.748039 11115 catalog_manager.cc:4573] TS e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f80db2e37bbc4716ad3a5200c61ee97b: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:11.815663 13185 raft_consensus.cc:1203] T 18b146cc4e164e1d9f9f827217c9b3ab P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:05:11.837716 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:11.840782 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:11.842031 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:11.842744 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:11.843557 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:11.927326 11268 raft_consensus.cc:1203] T f61e70ba02384f81b9ae36b1c65465b9 P e2dafc1866314669ad0de8fb7bf7fd05 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 08:05:11.955188 683 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:05:11.955329 683 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
W20240501 08:05:11.958921 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:11.959197 683 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
W20240501 08:05:11.960755 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:11.960847 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:05:11.962249 683 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:05:11.966449 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:11.966452 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:11.970490 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:11.979245 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:11.981277 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:11.982369 11661 raft_consensus.cc:1203] T b4707b952d174a9bb877ad8bd7a11268 P 603919bb02364c04b6c530245b8b02f2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 08:05:11.988797 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:12.000777 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:12.004907 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:12.019207 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:12.031566 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:12.032157 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:12.119047 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:12.122064 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:12.128345 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:12.128350 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:12.131237 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:12.225008 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:12.226907 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:12.228480 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:12.231302 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:12.232057 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:12.342192 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:12.342545 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:12.344102 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:12.346750 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:12.360467 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:12.385569 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:12.391995 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:12.392735 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:12.403452 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:12.412158 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:12.506709 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:12.507248 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:12.509323 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:12.509961 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:12.511921 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:12.540253 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:12.548816 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:05:12.621146 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:12.622788 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:12.624565 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:12.634984 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:05:12.637780 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:12.639496 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:12.668481 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:05:12.702147 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:12.703043 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:12.703612 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:12.705844 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:12.719359 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:12.732595 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:12.759464 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:12.793567 11115 catalog_manager.cc:4573] TS e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet f80db2e37bbc4716ad3a5200c61ee97b: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:05:12.929996 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:12.942291 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:12.945305 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:12.947613 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:12.950387 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:12.961326 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:05:12.961997 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:12.962648 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:13.053282 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:13.055501 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:13.056015 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:13.061733 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:13.068271 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:05:13.080638 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:13.093382 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:13.099308 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:13.099399 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:13.111665 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:13.120394 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:13.128888 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:05:13.162909 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:05:13.214206 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:13.214614 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:13.217834 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:13.218515 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:13.220196 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:13.317467 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:13.317541 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:13.319151 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:13.321295 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:13.333185 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:13.376632 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:13.403908 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:05:13.422113 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:13.428191 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:13.428995 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:13.439524 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:05:13.446707 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:13.455785 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:05:13.459540 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:13.475812 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:13.483378 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:13.493253 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:13.495988 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:13.502286 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:13.711949 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 08:05:13.738314 1396 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:05:13.738523 1396 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:05:13.739008 1396 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
W20240501 08:05:13.743738 13122 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:13.744203 13121 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:13.744278 13121 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9; no voters: 60e8857abe7c41038af46ebb5a095a90, ca177138f2c44d2eafa1646b795f7608
I20240501 08:05:13.744966 1396 raft_consensus.cc:2726] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:05:13.758038 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:13.758069 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:13.765070 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240501 08:05:13.775095 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:13.775251 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:13.794464 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:13.829665 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:13.843042 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:13.843295 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:13.844144 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:13.848670 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:13.855108 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:13.944736 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:13.947042 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:13.947876 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:13.947966 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:13.950356 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:13.956162 941 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:05:13.956292 941 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:05:13.956714 941 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
W20240501 08:05:13.957463 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 08:05:13.958113 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:13.958220 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:13.958259 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:05:13.959148 941 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:05:14.046007 11114 catalog_manager.cc:4573] TS 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c95c4f3ef08f44d1b69fb7b98426f5af: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:14.055469 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:14.056542 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:14.065385 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:14.069039 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:14.076061 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:14.122136 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240501 08:05:14.136356 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:14.137676 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:14.138662 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:14.148396 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:14.149240 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:14.180450 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:05:14.211833 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:14.215238 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:14.216495 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:14.217779 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:14.227181 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:14.320101 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:14.323733 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:14.324744 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:14.325837 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:14.326185 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:14.339022 11117 maintenance_manager.cc:419] P 2ed840e35a9f4afe9077d15510856d4e: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7332 bytes on disk
I20240501 08:05:14.341938 11116 maintenance_manager.cc:643] P 2ed840e35a9f4afe9077d15510856d4e: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.003s user 0.002s sys 0.000s Metrics: {"thread_start_us":137,"threads_started":2,"wal-append.queue_time_us":165}
W20240501 08:05:14.445973 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:14.448807 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:14.451505 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:14.452337 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:14.453431 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:14.468134 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:14.497066 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 16 similar messages]
W20240501 08:05:14.499996 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 08:05:14.628777 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:14.630693 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:14.630693 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:14.642655 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:14.647992 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:14.734390 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:14.735443 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:14.745263 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:14.745904 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:14.748724 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:14.795265 11115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 188 similar messages]
W20240501 08:05:14.810375 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:05:14.816447 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:14.820324 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:14.830050 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:14.838088 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:14.851178 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:14.891902 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:14.893986 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:14.895318 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:14.896334 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:14.897276 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:14.930882 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:14.993664 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:14.993747 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:14.997174 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:15.004142 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:15.015250 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:15.075452 11115 catalog_manager.cc:4573] TS e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ea97f34590684bcf998228245dbcd9b0: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:05:15.092746 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:05:15.101644 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:15.104956 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:15.105324 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:15.108386 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:15.110054 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:15.142819 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:05:15.189818 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:15.192536 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:15.198593 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:15.200976 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:15.214227 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:15.230561 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:15.257038 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:05:15.420745 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:15.424850 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:15.425282 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:15.433347 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:15.434273 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:15.464679 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:15.486176 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:05:15.530690 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:15.534104 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:15.542372 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:15.542441 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:15.543560 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:15.544631 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:15.570323 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:15.576853 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:15.581825 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:15.586010 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:15.587678 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:15.628182 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:15.654608 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:15.679605 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:05:15.687983 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:15.688544 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:15.692724 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:15.693576 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:15.698493 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:15.788306 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:15.790122 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:15.790746 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:15.796700 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:15.797029 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:15.820943 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:05:15.903102 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:15.916056 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:15.915853 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:15.916797 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:15.917474 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:15.924314 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:15.942260 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:15.947539 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:15.957751 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:15.960780 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:16.033067 1396 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:05:16.033218 1396 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:05:16.033655 1396 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
W20240501 08:05:16.034693 13122 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:16.034783 13121 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:16.034862 13121 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9; no voters: 60e8857abe7c41038af46ebb5a095a90, ca177138f2c44d2eafa1646b795f7608
I20240501 08:05:16.034991 1396 raft_consensus.cc:2726] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:05:16.062757 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 08:05:16.077489 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:16.176380 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:05:16.268267 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:16.279527 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:16.281065 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:16.282995 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:16.284791 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:16.297194 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:16.300725 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:16.318636 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:16.322579 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:16.328701 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:16.414700 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:16.415704 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:16.416503 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:16.421818 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:16.438481 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:16.443543 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240501 08:05:16.466224 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 08:05:16.521369 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:16.523684 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:16.525843 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:16.526592 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:16.528565 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:16.574194 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 08:05:16.628968 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:16.636760 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:16.647768 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:16.657868 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:16.658607 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:16.666337 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:16.668685 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:16.668699 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:16.668713 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:16.682021 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:16.686296 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:16.788215 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:16.788453 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:16.791123 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:16.795503 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:16.800170 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:16.811913 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:05:16.829897 941 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:05:16.830045 941 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:05:16.830472 941 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
W20240501 08:05:16.831670 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:16.831789 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:16.831825 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:05:16.831945 941 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:05:16.982014 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:16.983685 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:16.984383 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:16.986706 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:16.989420 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:17.027798 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:17.056969 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:17.098997 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:17.100787 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:17.102104 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:17.102808 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:17.115067 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:17.203509 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:17.204605 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:17.211237 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:17.217687 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:17.224684 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:17.263594 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:05:17.299489 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:17.302510 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:17.304147 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:17.309307 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:17.318593 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:17.363718 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:17.364179 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:17.366369 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:17.367420 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:17.371748 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:17.464218 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:17.469158 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:17.469373 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:17.470600 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:17.470968 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:17.523046 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:17.567198 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:17.567229 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:17.568756 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:17.569476 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:17.605194 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:17.638271 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:17.640736 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:17.646278 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:17.655012 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:17.662571 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:17.728190 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:17.732730 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:17.747959 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:17.808293 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:17.907307 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:05:17.922511 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:17.926577 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:17.926870 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:17.937068 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:17.943877 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:18.002880 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:18.004293 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:18.006203 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:18.012578 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:18.021096 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:18.038190 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:18.063469 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:18.064047 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:18.069572 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:18.072227 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:18.081571 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:05:18.081979 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:18.110574 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:18.158931 11114 catalog_manager.cc:4573] TS 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c95c4f3ef08f44d1b69fb7b98426f5af: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:18.168218 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:18.171003 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:18.171041 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:18.179805 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:18.182593 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:18.204097 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:18.266126 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:05:18.272641 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:18.274055 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:18.275393 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:18.277877 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:18.294412 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:18.363535 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:18.364243 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:18.365928 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:18.366331 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:18.387462 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:18.435588 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:18.435595 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:18.436396 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:18.437762 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:18.440351 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:18.481540 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:18.689486 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:18.690780 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:18.693373 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:18.700210 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:18.711375 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:18.760375 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:05:18.774516 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:18.776747 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:18.777118 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:18.779433 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:18.779847 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:18.882728 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:18.882728 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:18.883280 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:18.885118 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:18.886236 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:18.926702 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 08:05:18.930541 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 08:05:18.931149 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:19.000470 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:19.004173 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:19.005190 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:19.006337 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 08:05:19.010332 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 08:05:19.011147 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:19.032704 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:19.049034 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:19.057899 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:19.064376 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:19.073285 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:19.085436 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:19.129983 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240501 08:05:19.142149 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:19.147022 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:19.148764 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:19.148892 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:19.150274 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:19.163993 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240501 08:05:19.191275 11115 catalog_manager.cc:4573] TS e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ea97f34590684bcf998228245dbcd9b0: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:19.253661 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:19.253852 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:19.258826 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:19.258869 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:19.267674 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:19.341101 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:19.375399 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:19.376689 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:19.377396 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:19.379498 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:19.382923 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:19.567837 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:19.569893 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:19.573565 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:19.579989 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:19.587828 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:19.659842 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 08:05:19.677157 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:19.683887 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:19.684825 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:19.685338 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:19.695012 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:19.714084 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:05:19.737210 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:19.737661 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:19.749490 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:19.755369 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:19.755970 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:19.767750 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:19.837771 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:19.840975 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240501 08:05:19.842921 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:19.847744 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:19.847961 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:05:19.849009 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:19.849704 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:19.863155 11114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 08:05:19.942677 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:19.942775 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:19.944304 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:19.946545 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:19.964860 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:19.977721 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:05:20.055066 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:20.060429 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:20.064038 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:20.064586 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:20.064716 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:20.105793 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:20.106029 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:20.112838 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:20.113576 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:20.117894 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:20.133945 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:20.134711 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:20.229301 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:20.245483 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:20.382782 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:20.389621 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:05:20.410935 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:20.414090 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:20.418725 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:20.426335 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:20.426640 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:20.475682 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:20.480753 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:20.482386 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:05:20.484772 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:20.491106 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:20.511606 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:20.550333 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:20.550689 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:20.561089 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:20.561483 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:20.562202 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:20.655669 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:20.661715 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:20.662248 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:20.662830 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:20.665447 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:20.690495 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:20.762951 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:20.767563 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:20.770331 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:05:20.773744 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:20.779590 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:20.782205 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:20.856575 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:20.861655 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:05:20.861689 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:20.862563 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:20.866868 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:20.884152 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:20.930256 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:20.930744 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:20.936161 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:20.936534 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:20.937892 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:21.012548 3868 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:05:21.012692 3868 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:05:21.013080 3868 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
W20240501 08:05:21.014047 13121 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:05:21.014132 13122 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:21.014205 13122 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9; no voters: 60e8857abe7c41038af46ebb5a095a90, ca177138f2c44d2eafa1646b795f7608
I20240501 08:05:21.014330 3868 raft_consensus.cc:2726] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:05:21.183873 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:21.184379 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:21.189682 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:21.190582 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:21.204980 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:21.214891 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:21.260160 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 08:05:21.270336 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:21.274672 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:21.277350 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:21.279681 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:21.281951 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:21.291798 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:21.319480 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 08:05:21.332340 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240501 08:05:21.379166 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:21.384397 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:21.388195 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:21.391335 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:21.403374 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:21.510495 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:21.510632 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:21.513720 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:21.533489 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:21.534130 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:21.547462 3898 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:05:21.547592 3898 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:05:21.547973 3898 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
W20240501 08:05:21.552455 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:21.552606 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:21.552661 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:05:21.553264 3898 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:05:21.555964 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:21.556144 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:21.556556 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:21.557586 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:21.571061 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:21.661758 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:21.662618 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:21.662760 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:21.663288 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:21.664441 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:21.701036 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:21.742611 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240501 08:05:21.766233 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:21.766705 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:21.769325 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:21.771781 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:21.792486 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:21.815317 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:05:21.869868 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:21.870019 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:21.870038 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:21.871838 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:21.872432 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:22.083541 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:22.086354 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:22.087195 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:22.087810 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:22.097625 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:22.144464 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:22.192090 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:22.194692 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:22.196992 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:22.211046 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:22.214545 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:22.221499 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:22.222386 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:22.226122 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:22.229288 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:22.237691 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:22.269245 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:22.306960 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:05:22.345407 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:22.345968 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:22.347087 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:22.347493 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:22.354701 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:22.373456 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:22.411219 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:05:22.444015 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:22.445376 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:22.448110 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:22.448855 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:22.451784 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:22.553234 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:22.557572 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:22.559499 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:22.560248 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:22.577458 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:22.593466 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:22.597319 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:22.610128 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:22.615621 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:05:22.623891 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:22.625783 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:22.657869 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:22.712239 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:22.790138 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:22.831094 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:05:22.884732 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:22.886291 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:22.887101 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:22.889365 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:22.889443 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:22.955876 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:22.957501 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:22.959106 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:22.959264 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:22.960513 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:23.050037 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:23.050346 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:23.052770 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:23.053757 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:23.069046 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:23.083081 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:05:23.153543 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:23.158965 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:23.164992 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:23.165246 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:23.172781 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:23.200026 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:05:23.247213 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:23.247857 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:23.253929 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:23.254791 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:23.263002 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:23.282747 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:23.283510 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:23.303735 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:23.306974 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:23.308852 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:23.315563 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:23.316984 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:23.416720 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:23.419545 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:23.422245 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:23.425096 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:23.436607 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:23.576658 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:23.576969 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:23.581230 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:23.584282 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:23.592406 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:23.690250 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:05:23.704727 3868 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:05:23.704867 3868 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:05:23.705341 3868 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
W20240501 08:05:23.707060 13121 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:05:23.707803 13122 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:23.707881 13122 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9; no voters: 60e8857abe7c41038af46ebb5a095a90, ca177138f2c44d2eafa1646b795f7608
I20240501 08:05:23.708137 3868 raft_consensus.cc:2726] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:05:23.723011 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:23.723549 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:23.730587 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:23.732376 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:23.740954 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:23.829062 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:23.830502 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:23.831705 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:23.835271 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:23.835922 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:23.846333 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 08:05:23.882210 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:23.888979 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 08:05:23.944603 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:23.947571 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:23.951509 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:23.971261 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:23.982113 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:23.985574 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:23.986230 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:23.989313 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:23.993299 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 08:05:24.006345 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:24.008165 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:24.090260 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240501 08:05:24.092200 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:24.092464 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:24.096871 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:24.098387 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:24.106209 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:24.138628 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240501 08:05:24.202224 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:24.203322 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:24.206890 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240501 08:05:24.211772 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:24.214761 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:24.222604 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:24.295492 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:24.297770 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:24.298368 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:24.300171 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:24.302589 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:24.311780 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:24.516685 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:24.516901 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:24.520490 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:24.520875 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:24.527765 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:24.611663 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:24.612198 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:24.613062 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:24.619323 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:24.644825 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:24.666283 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:24.670246 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:24.678768 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:24.681133 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:24.694569 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:24.707348 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:05:24.708206 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:24.711472 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:24.744810 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:05:24.781008 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:24.781401 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:24.784848 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:24.785406 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:24.788048 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:24.876096 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:24.882740 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:24.884841 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:24.892127 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:24.893177 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:24.899643 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:24.975769 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:24.977846 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:24.983753 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:24.986153 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:24.986670 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:25.024309 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:05:25.038534 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:25.039152 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:25.043988 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:25.050504 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:05:25.052371 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:25.054775 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:25.117439 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:25.154834 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 14 similar messages]
W20240501 08:05:25.206394 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:25.216262 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:05:25.295800 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:25.296391 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:25.296933 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:25.298540 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:25.299993 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:25.395753 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:25.404654 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:25.404920 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:25.411811 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:25.412518 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:25.460129 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:25.500753 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:25.504930 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:25.510597 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:25.511576 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:25.516904 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:25.537585 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:05:25.616719 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:25.632658 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:25.638456 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:25.647529 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:25.648295 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:25.648833 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:25.651964 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:25.654748 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:25.657881 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:25.666581 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:25.743420 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:05:25.767900 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:25.769675 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:25.775918 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:25.779075 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:25.795396 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:25.822702 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:05:25.878943 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:25.881152 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:25.886633 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:25.888530 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:25.899088 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:25.961369 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:25.985071 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:25.986771 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:25.990193 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:25.998266 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:25.998345 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:26.052516 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:26.146243 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 08:05:26.166299 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 08:05:26.192082 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:26.193902 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:26.196367 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:26.208128 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:26.208204 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:26.304350 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:26.310166 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:26.310600 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:26.320533 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:26.324972 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:26.338869 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:26.342823 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:26.344895 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:26.347344 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:26.359823 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:26.381510 11114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:05:26.382297 11114 catalog_manager.cc:4573] TS 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c95c4f3ef08f44d1b69fb7b98426f5af: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:26.454691 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:26.460965 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:26.461261 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:26.461432 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:26.461618 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:26.551478 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:26.553705 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:26.557399 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:26.559036 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:26.571280 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:26.580914 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:26.581405 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240501 08:05:26.661038 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:26.663962 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:26.666697 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:26.676383 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:26.684929 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:26.715107 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:26.716460 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:26.719362 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:26.719748 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:26.725884 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:26.832474 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 08:05:26.857282 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240501 08:05:26.910537 5301 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:05:26.910679 5301 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:05:26.911082 5301 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
W20240501 08:05:26.912276 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:26.912772 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:26.912843 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:05:26.913143 5301 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:05:26.984251 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:26.988821 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:26.989799 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:26.992914 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:27.000469 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:27.018343 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:27.059062 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:27.061231 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:27.061378 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:27.063216 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:27.074381 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:27.141686 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:05:27.154920 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:27.155625 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:27.158041 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:27.160573 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:27.161895 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:27.202212 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:27.207862 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:05:27.265946 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:27.268841 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:27.269994 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:27.281447 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:27.282617 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:27.352486 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:05:27.357822 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:27.364933 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:27.366106 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:27.373332 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:27.387032 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:27.395625 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:27.409363 11115 catalog_manager.cc:4573] TS e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ea97f34590684bcf998228245dbcd9b0: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:27.432628 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:27.433571 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:27.435559 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:27.436142 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:27.438464 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:27.476128 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:27.517354 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:05:27.539803 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:27.540263 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:27.545786 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:27.546610 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:27.547374 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:27.628574 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:05:27.692749 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:27.694032 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:27.694274 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:27.697803 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:27.698093 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:27.809278 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:27.850116 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:27.857815 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:27.857901 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:27.863936 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:27.864722 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:27.899395 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:27.932576 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:05:27.956092 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:27.959582 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:27.960119 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:27.973256 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:27.982371 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:28.031785 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:28.034616 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:28.042126 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:28.055860 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:28.056442 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:28.115412 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:28.115738 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:28.116017 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:28.116408 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:28.118575 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:28.216349 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:28.219345 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:28.219861 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:28.224362 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:28.226608 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:28.317294 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:28.318773 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:28.320494 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:05:28.331864 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:28.332060 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:28.332360 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:28.346160 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:05:28.389143 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:28.389416 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:28.390461 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:28.400496 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:28.402177 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:28.417733 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:28.459192 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 08:05:28.503504 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:05:28.633864 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:28.634341 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:28.635593 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:28.648319 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:28.648833 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:28.655393 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 08:05:28.733834 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:28.740305 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:28.753904 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:28.759873 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:28.772888 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:28.790555 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:28.793430 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:28.808333 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:28.811138 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:28.816155 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:28.909327 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:28.917053 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:28.919260 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:28.920033 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:28.925006 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:28.928748 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:29.012080 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:29.013932 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:29.015239 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:29.019754 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:29.038172 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:29.087038 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240501 08:05:29.090502 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240501 08:05:29.107347 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:29.108016 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:29.109490 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:29.117641 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:29.118921 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:29.169832 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:29.174042 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:29.177047 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:29.177161 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:29.177699 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:29.362215 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:29.410965 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 08:05:29.437454 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:29.439774 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:29.441653 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:29.452997 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:29.458931 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:29.477141 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240501 08:05:29.485648 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:29.522046 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:29.523769 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:29.525267 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:29.525378 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:29.529737 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:29.593859 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:05:29.624293 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:29.626060 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:29.627152 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:29.629073 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:29.632885 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:05:29.644011 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:29.667495 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:05:29.731748 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:29.734798 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:29.735273 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:29.737884 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:29.748463 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:29.802911 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:29.804837 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:29.807492 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:29.828236 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:29.834950 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:29.873594 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:29.875347 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:29.879163 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:29.887683 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:29.889642 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:29.976409 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:29.977767 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:29.978188 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:29.981717 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:30.002871 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:30.005817 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240501 08:05:30.007283 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:30.053431 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:30.097631 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:05:30.129317 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:30.129834 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:30.130704 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:30.131489 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:30.135397 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:30.241161 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:05:30.288280 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:30.290277 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:30.291352 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:30.293429 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:30.294096 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:30.370448 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 08:05:30.381763 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:30.384478 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:30.392588 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:30.400116 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:30.420579 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:30.454846 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:30.467123 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:05:30.477470 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:30.478014 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:30.483723 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:30.496639 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:30.506275 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:30.553663 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:30.554546 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:30.556794 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:30.557811 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:30.567019 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:30.598556 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:05:30.653717 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:30.655328 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:30.656430 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:30.657639 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:30.657634 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:30.774660 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:30.776794 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:30.777020 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:30.777272 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:30.778699 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:30.844552 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:30.846891 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:30.851897 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:30.855105 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:30.856232 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:30.928707 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:30.995899 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:30.997452 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:31.060850 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 08:05:31.076337 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:05:31.096832 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:31.097431 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:31.098485 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:31.101399 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 08:05:31.101698 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:31.118633 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:31.194309 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:31.198591 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:31.211190 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:31.229885 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:31.236749 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:31.239786 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:31.244549 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:31.245124 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:31.250255 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:31.261699 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:31.360525 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:31.362391 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:31.373965 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:31.377315 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:31.390373 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:31.404974 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:31.484319 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:31.484319 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:31.486673 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:31.491611 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:31.505496 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:31.594204 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:31.596505 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:31.603335 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:31.609652 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:31.624684 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:31.644316 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:31.644608 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:31.648849 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:31.649241 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:31.658444 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:31.813453 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:31.833948 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240501 08:05:31.908296 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:31.924437 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:05:31.972263 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:31.976701 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:31.983074 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:05:31.986948 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:31.987453 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:31.990334 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:31.999477 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:32.000420 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:32.012481 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:32.017521 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:32.027689 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:32.040719 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:05:32.117898 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:32.119763 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:32.120736 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:32.122320 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:32.137200 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:32.225986 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
I20240501 08:05:32.230233 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:32.230490 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:32.233788 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:05:32.236838 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:32.239238 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:32.243801 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:32.282526 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:32.340926 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:32.341385 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:32.341650 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:32.344861 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:32.361120 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:32.371349 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:32.379935 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:32.383152 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:32.387012 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:32.400907 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:32.497584 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:32.501194 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:32.502544 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:32.502576 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:32.512089 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:32.589175 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:32.669941 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:05:32.688597 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:32.691718 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:32.692056 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:32.693459 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:32.695915 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:32.762338 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:05:32.815549 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:32.818033 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:32.819234 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:32.822633 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:32.837706 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:32.885448 6970 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:05:32.885586 6970 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:05:32.886008 6970 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
W20240501 08:05:32.887094 13122 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:32.887190 13121 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:32.887282 13121 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9; no voters: 60e8857abe7c41038af46ebb5a095a90, ca177138f2c44d2eafa1646b795f7608
I20240501 08:05:32.887409 6970 raft_consensus.cc:2726] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:05:32.905158 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:32.919178 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:32.921070 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:32.923343 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:32.937687 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:32.938616 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:33.024422 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:33.025712 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:33.026087 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:33.029990 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:33.038200 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:33.060000 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:05:33.088454 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:33.088863 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:33.089617 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:33.101707 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:33.104923 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:33.184196 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:33.190587 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:33.199151 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:33.201666 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:05:33.202610 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:33.203299 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:33.292982 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:33.306357 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:33.307353 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:33.310329 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:33.315696 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:33.337337 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:33.383354 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:33.384210 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:33.385392 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:33.386056 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:33.407248 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:33.423712 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:33.533049 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 08:05:33.567627 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:05:33.631695 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:33.633567 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:33.636945 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:05:33.638520 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:33.640304 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:33.644551 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 08:05:33.652851 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:33.730127 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:33.733856 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:33.734398 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:33.734581 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:33.755079 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:33.790973 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:33.792376 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:33.792471 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:33.804140 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:33.813480 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:33.902766 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:33.903056 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:33.904464 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:33.907284 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:33.908972 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:33.915221 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:34.011993 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:34.015594 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:34.020035 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:34.020681 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:34.022823 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:34.091922 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 08:05:34.118759 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:34.126943 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:34.130733 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:34.135178 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:34.135279 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:34.184000 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:34.188102 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:34.188119 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:34.198654 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:34.201442 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:34.294915 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240501 08:05:34.301108 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:34.357506 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:05:34.371294 7094 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:05:34.371423 7094 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:05:34.371786 7094 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
W20240501 08:05:34.372794 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:34.372884 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:34.372925 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:05:34.373059 7094 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:05:34.447794 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:34.449271 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:34.449931 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:34.459259 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:34.472635 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240501 08:05:34.476316 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:34.481621 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:34.481910 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 08:05:34.482960 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:34.500341 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:05:34.532078 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:34.532196 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:34.532375 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:34.535976 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:34.535985 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:34.636794 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:34.640785 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:34.640815 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:34.642606 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:34.644652 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:34.750627 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:34.758656 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:34.758738 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:34.759434 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:34.775880 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:34.828466 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:34.831764 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:34.832494 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:34.840478 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:34.844414 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:34.894608 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:34.894795 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:34.898200 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:34.898916 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:34.908967 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:34.954916 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:34.983675 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:05:35.006714 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:35.007665 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:35.009604 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:35.015463 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:35.015792 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:35.057745 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 08:05:35.148262 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:35.150040 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:35.150816 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:35.152312 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:35.157478 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:35.313185 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:05:35.317428 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:35.317698 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:35.317806 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:35.320086 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:35.322221 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:35.399677 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:05:35.423089 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:35.425371 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:35.438629 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:35.439823 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:35.453768 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:35.509544 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:35.512162 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:35.522344 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:35.524878 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:35.532794 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:35.535513 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:35.552877 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 08:05:35.584220 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:35.584220 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:35.587352 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:35.588542 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:35.617609 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:35.662034 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:05:35.692518 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:35.693792 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:35.696547 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:35.698822 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:35.702999 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:35.815024 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:35.817903 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:35.818001 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:35.826716 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:35.827878 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:35.858697 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:35.865070 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:05:35.872314 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:35.875108 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:35.882193 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:35.886513 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:35.887460 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:35.911430 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:35.959478 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 08:05:36.143464 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:36.143867 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:36.148344 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:36.148344 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:36.148725 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:36.169203 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:05:36.170941 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:36.231629 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:36.233240 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:36.235603 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:36.239665 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:36.240463 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:36.289600 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:36.292152 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:36.292397 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:36.292827 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:36.310721 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:36.387130 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 08:05:36.394292 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:36.397696 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:36.404029 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:36.405217 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:36.418658 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:36.506281 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:36.507170 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:36.507483 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:36.508069 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:36.509797 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:36.589766 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:36.590533 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:36.594472 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:36.594573 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:36.601430 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:36.603291 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:36.630395 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:05:36.662523 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:36.666095 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:36.670329 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:36.674629 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:36.675905 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:36.764791 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 08:05:36.874082 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:36.882892 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 08:05:36.892037 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:05:36.902778 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:36.902880 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:36.906245 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:36.918095 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:36.925724 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:36.999902 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:37.000351 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:37.002067 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:37.004078 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:37.006707 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:37.032694 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:37.101971 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:37.103787 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:37.106499 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:37.115303 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:37.117125 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:37.203893 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 08:05:37.218040 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:37.221217 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:37.221307 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:05:37.226583 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:37.228307 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:37.249436 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:37.274227 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:37.276630 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:37.278884 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:37.283530 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:37.295652 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:05:37.297351 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:37.367476 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:37.372242 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:37.372681 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:37.375595 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:37.387790 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:37.461529 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:37.483752 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:37.484380 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:37.487004 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:37.487668 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:37.494644 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:37.583458 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:37.585996 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:37.586711 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:37.586999 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:37.589654 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:37.728178 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:05:37.791556 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:37.791651 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:37.795421 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:37.815716 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:37.817804 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:37.903621 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:37.905907 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:37.913870 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:37.916580 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:37.917194 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:37.963872 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:37.968194 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:37.976029 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:37.984228 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:37.992412 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:38.018937 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:38.022536 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:38.077386 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:38.081368 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:38.082134 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:38.082340 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:38.082695 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:38.083562 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:38.116981 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:05:38.181465 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:38.182354 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:38.182345 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:38.185485 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:38.190634 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:38.282997 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:38.290206 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:38.294610 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:38.296264 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:38.311811 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:38.343827 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:38.347754 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:38.351516 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:38.358683 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:38.365880 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:38.367228 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:38.451968 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:38.475996 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 08:05:38.521463 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:05:38.612449 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:38.616103 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:38.617659 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:38.617887 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:38.629739 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:38.673580 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:38.693464 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:38.693724 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:38.695672 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:38.697007 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:38.715390 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:38.794942 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:38.798482 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:38.805430 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:38.807533 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:38.809242 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:38.915493 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:38.918287 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:38.927567 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:38.936477 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 08:05:38.941829 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:38.958089 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:38.967916 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:38.976348 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:38.977126 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:38.982854 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:38.991623 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:39.055711 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:05:39.070740 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:39.076355 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:39.079284 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:39.081179 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:39.081445 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:39.132402 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 08:05:39.185371 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:39.187167 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:39.189018 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:39.191980 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:39.196374 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:39.217671 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 08:05:39.226145 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:39.226871 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240501 08:05:39.288753 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:39.289966 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:39.292433 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:39.306350 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:39.307740 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:39.355162 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:05:39.498942 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:39.502749 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:39.505128 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:39.505637 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:39.507509 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:39.591907 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:05:39.613945 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:39.618321 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 08:05:39.619431 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:39.625141 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:39.625638 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:39.633157 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:05:39.635409 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:39.636514 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:39.647079 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:39.647293 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:39.650750 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:39.674412 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:39.676997 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:39.766849 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:39.773133 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:39.773362 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:39.773795 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:39.774639 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:39.826922 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:05:39.879209 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:39.880383 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:39.882267 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:39.882759 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:39.890609 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:39.999706 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:40.002512 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:40.005079 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:40.011401 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:40.014822 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:40.024307 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:40.027614 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:40.028112 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:40.034124 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:40.064435 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:40.096134 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:05:40.096686 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:40.240505 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:40.337617 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:40.338296 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:40.339409 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:40.360868 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:40.361779 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:40.383193 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:40.383203 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:40.404678 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:40.406193 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:40.421425 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:40.503377 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:40.506901 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:40.507858 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:40.508908 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:40.510576 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:40.613063 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:40.613248 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:40.626204 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:40.626569 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:40.647693 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:40.651476 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:40.714445 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 08:05:40.718304 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:40.722106 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:40.723353 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:40.723384 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:40.723769 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:40.776523 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:40.777642 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:05:40.780081 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:40.782083 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:40.789578 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:40.792395 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:40.797850 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:40.820996 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:40.873725 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:05:40.898291 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:40.900439 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:05:40.903086 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:40.906819 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:40.907632 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:40.926601 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:41.013480 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:05:41.048902 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:41.050376 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:41.051090 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:41.059255 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:41.062198 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:41.115398 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 08:05:41.150238 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 08:05:41.217025 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:41.217519 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:41.222177 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:41.223918 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:41.228118 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:41.315491 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:41.318686 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:41.330328 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:41.332754 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:41.333070 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:41.357859 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:41.395288 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:41.403157 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:41.410701 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:41.413296 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:41.425287 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:41.474440 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:41.476542 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:41.476790 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:41.479525 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:41.489833 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240501 08:05:41.496937 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:41.591337 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:41.592871 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:41.593731 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:41.595477 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:41.619395 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:41.706502 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:05:41.714092 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:41.720222 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:41.720739 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:41.721136 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:41.738211 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:41.751771 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240501 08:05:41.762791 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:41.768507 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:41.772697 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:41.778627 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:41.796274 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:41.802281 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:41.801826 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:05:42.076614 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:42.077795 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:42.079577 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:42.089625 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:42.091948 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:05:42.098961 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:42.124773 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:42.127077 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:42.129156 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:42.148990 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:42.148990 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:42.193765 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:42.227098 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:42.227983 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:42.230232 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:42.230473 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:42.231513 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:42.309284 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:05:42.329890 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:42.330879 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:42.331173 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:05:42.333343 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:42.340898 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:42.345593 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:42.449610 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:42.450629 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:42.453653 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:42.454588 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:05:42.455910 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:42.470888 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:42.474573 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:42.479944 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:42.482112 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:42.501735 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:42.509241 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:42.583618 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:42.584204 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:42.586364 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:42.586840 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:42.587361 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:42.613999 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:42.761435 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:42.810564 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:42.814481 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:42.817098 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:42.817521 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:42.821506 11114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 08:05:42.830358 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:42.831733 11114 catalog_manager.cc:4573] TS 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c95c4f3ef08f44d1b69fb7b98426f5af: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:05:42.909451 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:42.910364 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:42.918977 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:42.919380 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:42.927196 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:43.015960 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:43.018626 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:43.019557 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:43.020592 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:43.030524 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:43.145120 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:43.145165 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:43.150292 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:43.154450 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:43.160230 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:43.171528 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:43.172366 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:05:43.178403 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:43.181859 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:43.185349 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:43.217479 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:43.242805 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:43.265717 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:43.285482 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:43.296605 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:43.299516 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:43.304154 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:43.310878 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:43.311079 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:43.325711 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 08:05:43.343434 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:05:43.400501 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:43.402845 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:43.407123 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:43.419057 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:43.428200 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:43.477974 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:05:43.497068 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:43.502053 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:43.503332 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:05:43.504912 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:43.515318 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:43.516139 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:43.591460 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 08:05:43.602972 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 08:05:43.717450 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:43.719640 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:43.720486 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:43.721071 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:43.721850 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:43.838725 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:43.841894 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:43.843847 11115 catalog_manager.cc:4573] TS e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ea97f34590684bcf998228245dbcd9b0: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:43.844429 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:43.847545 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:43.848258 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:43.851266 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:43.861076 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:43.861521 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:43.865394 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:43.899901 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:43.905280 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:43.988070 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:43.991279 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:43.993180 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:43.993891 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:44.011442 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:44.086458 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:05:44.092002 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:44.092775 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:44.093731 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:44.094792 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:44.104245 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:44.178279 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 08:05:44.199950 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:44.203521 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:44.204619 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:44.205268 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:44.207881 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:44.250397 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:44.256791 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:44.258915 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:44.268718 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:44.271606 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:44.333191 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:44.340458 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 08:05:44.494279 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:44.537315 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:44.537997 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:44.542760 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:44.546944 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:44.547459 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:44.599757 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:44.603060 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:44.603919 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:44.609814 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:44.625928 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:44.654327 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240501 08:05:44.702152 12600 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:05:44.702288 12600 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:05:44.702389 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:44.702711 12600 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
I20240501 08:05:44.702948 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:44.703922 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:44.704052 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:44.704102 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:05:44.704222 12600 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:05:44.706202 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:44.706698 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:44.715852 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:05:44.723012 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:44.814146 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:44.820686 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:44.824040 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:44.826841 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:44.829819 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:44.875759 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:05:44.901401 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:44.901502 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:44.905756 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:44.905836 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:44.927513 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:44.954079 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:44.966049 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:44.969898 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:44.973490 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:44.973896 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:44.974095 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:45.077204 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:05:45.078513 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:45.081315 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:45.083016 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:45.083056 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:45.101680 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:45.141250 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 08:05:45.236737 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:45.236855 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:45.238796 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:45.246666 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:45.247076 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:45.282460 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:05:45.391371 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:45.393350 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:45.393922 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:45.396361 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:45.416913 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:45.459407 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:45.498209 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:45.498353 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:45.498647 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:45.522825 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:45.524854 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:45.592597 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:45.594544 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:45.596411 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:45.607203 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:45.623577 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:45.668658 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:45.671599 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:45.678668 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:45.680814 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:45.683336 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:45.712880 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:45.721810 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:45.738729 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:05:45.772531 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:45.776306 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:45.776679 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:45.778789 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:45.786617 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:45.863248 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:05:45.890756 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:45.890821 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:45.891024 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:45.892243 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:45.912143 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:45.933743 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:45.969878 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:45.970248 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:45.972265 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:45.983814 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:45.984061 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:46.006487 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:05:46.046226 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:46.070261 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 08:05:46.083258 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:05:46.201623 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:46.201925 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:46.203567 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:46.204375 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:46.221132 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:46.239856 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 08:05:46.260208 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 08:05:46.308248 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:46.311102 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:46.320026 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:46.320101 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:46.338649 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:46.338843 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:46.347812 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:46.348573 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:46.361757 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:46.382776 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:46.465718 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:46.467764 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:46.468133 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:46.469110 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:46.479529 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:46.541606 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:46.562516 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:46.565416 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:46.566972 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:46.571069 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:46.578929 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:46.602660 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:05:46.685460 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:46.687489 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:46.688683 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:46.694671 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:46.695199 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:46.700125 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:46.711863 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:46.714099 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:46.737039 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:46.752177 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:46.918227 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:46.925124 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:05:47.022223 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:47.029127 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:47.029346 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:47.031193 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:47.031446 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:47.033579 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240501 08:05:47.087644 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:47.090257 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:47.098533 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:47.098985 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:47.101317 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:47.187984 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:47.189646 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:47.191201 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:47.195130 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:05:47.195236 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:47.198402 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:47.300053 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:47.300758 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:47.306974 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:47.310133 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:05:47.310956 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:47.319764 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:47.382759 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:47.385195 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:47.391790 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:47.392098 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:47.404922 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:47.437537 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:47.461865 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:47.462702 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:47.466612 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:47.470577 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:47.476989 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:47.483577 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:05:47.573264 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:47.573982 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:47.574226 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:47.575031 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:47.586632 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:47.635946 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 08:05:47.697281 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:47.698910 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:47.698992 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:47.707473 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:47.713930 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:47.865113 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:47.878329 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:47.878743 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:47.882514 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:47.882808 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:47.895146 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:47.995587 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:47.999238 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:48.009218 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:48.010264 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:48.011006 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:48.057731 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:48.064317 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:48.064421 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:48.065833 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:48.066103 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:48.080863 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:05:48.140661 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:48.141203 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:48.142124 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:48.144201 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:48.145185 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:48.145731 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:48.173034 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:05:48.238551 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:48.240427 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:48.241688 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:48.242221 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:48.253202 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:05:48.253885 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:48.292516 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:48.352833 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:48.353786 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:48.353919 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:48.357841 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:48.371155 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:05:48.371852 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:48.396235 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:05:48.405505 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:48.409121 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:48.426352 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:48.427829 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:48.435068 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:48.613142 12156 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:05:48.613279 12156 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:05:48.613801 12156 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
W20240501 08:05:48.614987 13121 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:05:48.615098 13122 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:48.615173 13122 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9; no voters: 60e8857abe7c41038af46ebb5a095a90, ca177138f2c44d2eafa1646b795f7608
I20240501 08:05:48.615304 12156 raft_consensus.cc:2726] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:05:48.691015 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:48.696926 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:48.698830 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:48.709298 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:48.710072 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:48.713291 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:48.719874 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 08:05:48.757967 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240501 08:05:48.775529 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:48.780151 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:48.781196 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:48.786095 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:48.789489 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 08:05:48.790109 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:48.847440 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:48.848150 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:48.849289 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:48.850483 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:48.851897 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:48.855417 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:05:48.871599 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:48.957145 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:48.958873 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:48.959506 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:48.962236 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:48.962738 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:49.065994 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:49.066911 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:49.069546 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:49.069862 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:49.073068 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:49.130753 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:49.136472 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:49.140370 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:49.152366 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:49.162369 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:49.201210 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 08:05:49.218952 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:49.221580 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:49.222627 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:49.222787 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:49.229351 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:49.392882 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:49.409040 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 08:05:49.442551 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:05:49.469947 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:49.472903 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:49.473776 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:49.474529 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:49.479713 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:49.553817 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:49.556227 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:49.556515 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:49.556953 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:49.567680 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:49.575170 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:05:49.655392 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:49.655998 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:49.659147 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:49.664582 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:49.668740 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:49.767488 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:49.773607 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:49.773752 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:49.774400 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:49.788264 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:49.830513 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 08:05:49.838310 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:49.840190 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:49.842229 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:49.854816 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:49.862555 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:49.907717 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:49.910656 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:49.913594 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:49.916755 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:49.919121 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:49.948781 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:50.016750 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:50.020859 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:50.022928 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:50.023572 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:50.024163 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:50.092368 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:05:50.161674 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:50.162379 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:05:50.163059 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:50.163580 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:50.165478 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:50.168975 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:50.216537 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 08:05:50.320549 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:50.322764 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:50.325547 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:50.334998 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:50.335345 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:50.385612 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:50.426261 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:50.429319 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:50.431092 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:50.436976 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:50.443249 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:50.500520 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:05:50.506474 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:50.509801 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:50.510522 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:50.511821 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:50.514858 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:50.543216 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:05:50.573024 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:50.575875 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:50.577572 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:50.586581 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:50.589040 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:50.603588 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:05:50.678597 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:50.682747 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:50.683774 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:50.684211 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:50.686301 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:50.794512 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:50.794755 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:50.795944 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:50.796128 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:50.799077 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:50.809104 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:50.822208 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:05:50.866873 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:50.867944 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:50.868207 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:50.868836 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:50.870218 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:50.929041 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:05:50.969318 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 08:05:51.025918 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 08:05:51.105078 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:51.105787 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:51.108615 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:51.109082 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:51.125001 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:51.210816 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:05:51.224972 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:51.225111 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:51.227695 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:51.227854 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:51.232368 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
I20240501 08:05:51.244565 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:51.252920 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:51.253420 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:51.265043 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:51.266714 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:51.277805 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:51.293610 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:51.372256 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:51.373811 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:51.378806 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:51.385854 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:51.387048 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:51.411991 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:05:51.417637 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:05:51.479180 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:51.479147 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 08:05:51.483415 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:51.486044 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:51.487915 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:51.493050 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:51.609846 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:51.611169 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:51.615845 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:51.616793 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:51.627624 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:51.648761 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:51.661651 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:51.668977 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:51.675063 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:51.676971 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:51.893136 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:51.893780 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:05:51.919831 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240501 08:05:51.941310 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:51.945344 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:51.950676 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:51.966539 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:51.978484 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:52.019603 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:52.022080 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:52.025456 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:52.025830 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:52.026613 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:52.119014 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:52.119688 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:52.120218 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:52.122525 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:52.141165 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:52.242628 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:52.245183 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:52.248416 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:52.254633 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:52.260990 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:52.272874 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:05:52.306906 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:52.308605 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:52.313133 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:52.314678 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:52.324895 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:52.349064 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:52.378121 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:52.382750 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:52.383587 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:05:52.384006 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:52.388653 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:52.391301 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:52.486665 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:52.488880 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:52.490581 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:52.491204 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:52.492930 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:52.624233 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:52.627416 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:52.628854 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:52.629506 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:52.629966 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:52.664315 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:05:52.790164 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:52.793181 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:52.802783 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:52.805636 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:52.819286 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:52.888335 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 08:05:52.889000 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:05:52.903918 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:52.909538 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:52.909906 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:52.910802 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:52.919982 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:52.927598 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:52.981357 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:52.985554 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:52.985554 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:52.993552 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:53.011142 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:53.053745 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:53.057386 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:53.058889 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:53.062597 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:53.067461 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:53.075852 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:05:53.156177 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:53.160107 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:53.160598 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:53.161737 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:53.164014 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:53.169651 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:05:53.188889 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:05:53.272624 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:53.276901 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:53.278345 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:53.285319 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:53.296273 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:53.304788 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:53.341771 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:53.342253 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:53.342331 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:53.346701 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:53.359280 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:53.508103 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 08:05:53.514993 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:05:53.590071 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:53.592478 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:53.592610 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:53.592645 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:53.611580 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:53.629611 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:05:53.682543 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:53.683840 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:53.686365 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:53.686882 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:53.687861 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:53.749931 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:53.752584 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:53.752827 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:53.755244 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:53.769687 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:53.863638 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:53.863929 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:53.864066 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:53.864215 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:53.874890 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
I20240501 08:05:53.877480 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:53.898044 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:53.966543 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:05:53.968964 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:53.970716 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:53.979223 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:05:53.982224 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:53.982224 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:53.993054 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:54.007313 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 08:05:54.033535 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:54.033937 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:54.034086 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:54.042613 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:54.067873 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:54.117944 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:54.118896 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:54.119668 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:54.120273 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:54.120522 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:54.263937 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 08:05:54.319648 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:05:54.351868 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:54.353802 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:54.356205 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:54.359799 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:54.362586 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:54.413250 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240501 08:05:54.446733 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:54.447511 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:54.450882 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:54.453063 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:54.464998 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:54.508559 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:54.542930 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:54.548215 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:54.551944 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:54.558941 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:54.559159 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:54.674970 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:54.675526 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:54.675523 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:54.688750 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:54.692081 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:05:54.694317 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:54.699684 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:54.716228 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:54.737495 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:54.737519 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:54.737541 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:54.825569 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:54.826824 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:54.830879 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:54.838645 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:54.841226 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:54.874742 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 08:05:54.905761 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:54.940026 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:54.940145 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:54.940439 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:54.952157 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:54.954627 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:55.017994 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:05:55.034619 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:55.036973 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:55.037778 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:55.039055 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:05:55.039968 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:55.048725 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:55.249104 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:55.250908 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:55.254930 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:55.259362 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:55.271888 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:55.283838 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:05:55.349422 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:55.351492 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:55.355722 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:55.355876 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:55.358284 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:55.417940 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:55.422034 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:55.423708 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:55.424952 17403 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:05:55.425067 17403 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:05:55.425375 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:55.425479 17403 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
W20240501 08:05:55.426964 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 08:05:55.427426 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:55.427544 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:55.427588 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:05:55.427807 17403 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:05:55.431149 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:55.473934 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:05:55.522099 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:55.527945 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:55.530808 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:55.531378 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:55.534101 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:55.535495 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:55.631664 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:55.632844 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:55.641521 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:55.644161 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:55.645418 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:55.709105 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:55.711525 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:55.713068 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:55.715178 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:55.719610 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:55.762291 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:05:55.790691 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:55.792997 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:55.795573 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:55.797127 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:55.800467 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:55.814801 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:05:55.815574 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:05:55.899343 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:05:55.960230 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:05:56.021212 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:56.022662 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:56.023007 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:56.031899 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:56.032088 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:56.035609 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:56.120893 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:56.121778 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:56.124665 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:56.129273 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:56.131543 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:56.182071 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 08:05:56.224757 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:56.224783 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:56.224785 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:56.232053 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:56.235088 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240501 08:05:56.248687 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:56.349454 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:56.357203 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:56.359074 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:56.363396 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:56.367931 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:56.371222 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:56.373335 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:56.389217 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:56.393385 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:56.395133 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:56.495532 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:56.496932 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:56.501009 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:56.505789 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:56.507537 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:56.515991 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 08:05:56.519644 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:56.603035 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:56.604244 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:56.605432 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:56.606918 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:56.621399 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:56.683683 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 08:05:56.700232 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:56.702447 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:56.702888 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:56.703722 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:56.705391 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:56.772181 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:05:56.870676 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:05:56.913311 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:56.914775 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:56.914849 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:56.920331 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:56.925004 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:57.014896 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 08:05:57.022490 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:57.022665 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:57.030695 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:05:57.031503 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:57.034664 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:57.035638 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:57.036314 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:57.050930 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:57.051982 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:57.055158 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:57.057061 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:57.085381 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:57.163347 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:57.165432 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:57.165921 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:57.168017 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:57.178963 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:57.263289 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:57.265739 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:57.268893 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:57.273042 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:57.278283 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:57.388995 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:05:57.394918 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:57.395207 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:57.399348 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:57.409024 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:57.409375 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:57.432102 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:57.432902 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:57.441068 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:57.441452 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:57.447502 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:57.546607 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:57.620409 18390 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:05:57.620560 18390 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:05:57.620975 18390 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
W20240501 08:05:57.622187 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:05:57.622320 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:05:57.622362 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:05:57.622582 18390 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:05:57.704469 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:05:57.765072 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:57.768455 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:57.781448 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:57.783319 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:57.787472 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:57.798610 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:57.814152 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:57.825425 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:57.826408 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:57.833101 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:57.864253 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:05:57.919728 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:57.928189 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:57.928810 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:57.934517 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:57.943022 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:58.009145 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:58.037843 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:58.040021 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:58.040330 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:58.043303 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:58.050443 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:58.061208 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:05:58.063818 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:05:58.144091 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:58.144673 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:58.147480 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:58.153005 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:58.156404 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:58.163295 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:58.165748 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:58.186905 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:58.187084 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:58.196843 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:58.296567 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:58.296689 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:05:58.298202 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:58.298728 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:58.299419 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:58.304122 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:58.416169 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:05:58.478541 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:05:58.500041 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:58.500968 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:05:58.501842 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:58.503342 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:58.506719 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:58.522512 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:58.589609 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:58.603020 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:58.603103 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:58.603508 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:58.605544 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:58.619292 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:05:58.635759 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:05:58.701357 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:58.706843 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:58.706950 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:58.708564 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:58.709765 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:58.714709 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240501 08:05:58.837603 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:58.841499 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:58.848453 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:58.848948 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:58.849428 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:58.869982 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:58.871457 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:58.881584 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:58.886493 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:58.888351 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:05:58.930712 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 08:05:58.988013 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:58.989706 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:58.990077 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:58.990841 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:58.992038 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:59.080175 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:59.091393 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:59.092106 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:59.092479 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:59.106139 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:59.108172 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:59.195461 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:59.195717 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:59.196359 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:59.207484 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:59.208912 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:05:59.266103 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240501 08:05:59.301923 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 08:05:59.314395 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 08:05:59.354820 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:05:59.403389 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:59.406412 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:59.406575 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:59.408802 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:59.429805 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:59.496032 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:05:59.520556 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:59.521245 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:05:59.527103 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:05:59.527560 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:59.529814 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:59.540030 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:05:59.555548 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:05:59.571543 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:59.574458 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:59.575906 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:59.578528 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:59.578550 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:59.690641 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:59.690845 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:59.691273 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:59.696969 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:59.711655 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:59.797277 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:59.800112 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:59.804962 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:59.819563 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:59.835969 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:59.882133 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:05:59.886039 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:59.887975 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:59.889447 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:59.912228 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:59.977303 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:05:59.979641 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:05:59.979796 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:05:59.980006 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:05:59.980773 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:00.062191 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:00.102819 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 08:06:00.103420 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:00.207721 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:00.210120 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:00.214761 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:00.215608 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:00.239092 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:00.292447 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:06:00.309204 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:00.311326 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:00.313457 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:00.320382 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:00.326475 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:00.413975 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:00.414602 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:00.414829 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:00.417666 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:00.427090 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:00.451820 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:06:00.497418 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:06:00.543514 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:00.544257 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:00.544685 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:00.545030 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:00.550879 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:00.552757 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:06:00.552847 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:00.567013 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:00.587762 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:00.590631 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:00.592921 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:00.674278 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:00.675024 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:00.679409 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:00.686470 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:00.688990 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:00.690325 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:00.703029 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:06:00.780030 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:00.782963 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:00.784030 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:00.784853 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:00.804015 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:00.831557 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:00.864619 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:06:00.905887 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:00.909106 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:00.909132 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:00.912114 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:00.935535 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:01.092815 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:01.095927 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:01.096386 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:01.099257 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:01.110046 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:01.111212 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:01.120774 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:01.142088 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:06:01.197543 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:01.201289 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:01.207660 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:01.210000 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:01.220687 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:01.283879 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:01.286839 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:01.289484 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:01.292737 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:01.296370 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:01.337358 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 08:06:01.357403 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:01.363128 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:01.366477 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:01.370299 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:01.373169 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:01.430516 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240501 08:06:01.475436 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:01.476747 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:01.479718 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:01.480991 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:01.483603 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:01.496086 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:01.562625 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:06:01.591563 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:01.594684 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 08:06:01.594817 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:01.601622 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:01.603446 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:01.618746 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:01.642881 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:01.649848 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:01.652933 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:01.654949 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:01.655416 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:01.812332 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 08:06:01.924122 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240501 08:06:01.935928 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:01.939576 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:01.946199 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:06:01.947026 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:01.949761 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:01.950655 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:02.022141 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:02.024399 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:02.024489 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:02.031431 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:02.032346 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:02.048274 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:02.090032 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:02.091627 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:02.092756 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:02.093329 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:02.098970 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:02.176419 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:06:02.182443 17933 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:06:02.182595 17933 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:06:02.183084 17933 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
W20240501 08:06:02.186992 13122 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:02.187608 13121 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:06:02.187693 13121 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9; no voters: 60e8857abe7c41038af46ebb5a095a90, ca177138f2c44d2eafa1646b795f7608
I20240501 08:06:02.187898 17933 raft_consensus.cc:2726] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:06:02.200932 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:02.204871 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:02.216089 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:02.219280 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:02.219729 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:02.323024 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:02.323294 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:02.324553 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:02.327204 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:02.334275 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:02.346518 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:06:02.396950 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:02.402652 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:02.405236 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:02.407879 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:02.413134 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:02.476991 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:02.478695 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:02.482576 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:02.490618 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:02.497488 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:02.631242 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:02.635716 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:02.719951 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:02.722754 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:02.723670 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:02.725036 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:02.734824 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:02.826321 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:02.827142 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:02.829849 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:02.840876 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:02.843344 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:02.852061 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:06:02.931488 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:02.931643 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:02.933892 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:02.936899 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:02.948225 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:03.045406 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:03.048586 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:03.056510 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:03.070781 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:03.073877 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:03.077214 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:03.080247 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:03.084213 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:03.105316 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:03.106797 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:03.192008 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:06:03.192874 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:06:03.195408 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:03.195649 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:03.195712 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:03.198324 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:03.202144 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:03.213240 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:03.238272 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:06:03.301784 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:03.303856 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:03.304560 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:03.314378 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:03.316380 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:03.370707 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:06:03.401751 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:03.402537 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:03.415033 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:03.415829 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:03.417127 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:03.514997 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:06:03.611606 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:03.614106 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:03.614683 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:03.621013 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:03.624294 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:03.687979 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:03.732846 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:03.733445 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:03.733796 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:03.734210 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:03.735316 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240501 08:06:03.736726 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:06:03.737287 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:03.744762 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:03.758771 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:03.762681 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:03.768878 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:03.786772 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:03.801275 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 08:06:03.855978 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:06:03.884164 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:03.884214 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:03.886595 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:03.887810 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:03.888620 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:03.987804 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:03.991608 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:03.991956 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:03.998523 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:04.006829 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:04.103698 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:04.103698 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:04.110141 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:04.114403 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:04.120090 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:04.122367 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:04.122694 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240501 08:06:04.122808 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:04.130599 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:04.139700 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:04.153115 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:06:04.172374 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:04.178826 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:04.229979 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:04.291582 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:06:04.415414 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:04.445966 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 08:06:04.461921 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:04.465914 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:04.469738 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:04.470352 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:04.471540 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:04.506319 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:04.516721 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:04.520439 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:04.522225 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:04.523420 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:04.597036 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:04.600404 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:04.601334 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:04.603833 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:04.604898 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:04.660095 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:04.713578 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:04.717101 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:04.717738 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:04.719305 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:04.726624 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:04.836351 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:04.837035 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:04.845922 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:04.847379 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:04.857250 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:04.869132 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:04.886260 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:04.887040 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:04.894590 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:04.895073 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:04.966584 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:06:04.970481 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:04.973467 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:04.973558 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:04.976447 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:04.976969 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:05.021574 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:05.067555 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:05.103578 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:06:05.198724 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:05.199265 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:05.201670 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:05.210992 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:05.226300 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:05.300254 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:05.301416 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:05.305579 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:05.306977 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:05.310578 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:05.403947 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:05.404522 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:05.408165 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:05.410085 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:05.411756 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:05.438747 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 08:06:05.494882 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:05.500140 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:06:05.503166 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:06:05.531231 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:05.533336 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:05.537611 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:05.539808 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:05.555408 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:05.558466 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:05.560930 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:05.569114 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:05.569320 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:05.594822 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:05.665760 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:05.667366 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:05.667948 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:05.669983 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:05.682305 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:05.776739 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:05.777177 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:05.777709 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:05.778604 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:05.780258 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:05.841572 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:06:05.887068 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:05.888124 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:05.890658 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:05.897501 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:05.899540 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:06.011332 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:06.033457 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:06.041038 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:06:06.091746 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:06.093787 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:06.098618 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:06.100066 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:06.106813 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:06.109225 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 08:06:06.141028 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:06:06.203189 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:06.214643 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:06.214758 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:06.216352 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:06.225656 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:06.234642 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:06.237058 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:06.251075 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:06.252153 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:06.270511 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:06.330654 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:06:06.360781 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:06.361086 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:06.362133 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:06.364449 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:06.373804 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:06.427439 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240501 08:06:06.459508 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:06.464104 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:06.466223 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:06.466647 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:06.478591 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:06.520953 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:06:06.577376 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:06.580127 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:06.585191 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:06:06.585825 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:06.589579 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:06.589630 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:06.601480 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:06.607374 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:06.617533 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:06.618034 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:06.620215 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:06.635468 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:06.743335 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240501 08:06:06.904426 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 08:06:06.942596 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:06.943482 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:06.947628 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:06.948233 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:06.962796 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:06.973102 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:06:07.003057 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:07.003268 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:07.003360 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:07.008877 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:07.008955 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:07.045640 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:07.074754 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20240501 08:06:07.075387 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:07.075928 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:07.080710 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:07.083604 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:07.087499 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:07.177687 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:07.181644 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:07.183339 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:07.189882 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:07.198359 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:07.296870 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:07.303213 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:07.303254 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:07.332120 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:07.332504 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:07.345695 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:07.350837 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:07.358686 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:07.385684 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:07.386551 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:07.463110 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:07.464493 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:07.464493 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:07.465106 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:07.465422 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:07.485857 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:07.591006 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:07.655936 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:06:07.679874 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:07.680284 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:07.685035 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:07.686632 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:07.687611 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:07.689162 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:06:07.778158 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:07.778838 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:07.789398 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:07.789659 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:07.790082 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:07.882205 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:07.883524 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:07.894281 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:07.894511 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:07.895552 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:08.011619 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:08.012674 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:08.014022 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:08.019075 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:08.019318 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:08.021989 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:08.022971 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:08.029698 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:08.035831 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:08.038465 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:08.064942 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:06:08.074010 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:08.147150 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:08.147799 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:08.148928 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:08.149659 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:08.152144 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:08.254040 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:08.254860 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:08.258881 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:08.263298 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:08.268054 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:06:08.269382 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:08.315353 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:06:08.363153 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:08.363637 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:08.366288 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:08.367404 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:08.370177 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:08.371762 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:08.521883 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:06:08.563625 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:08.565208 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:08.567893 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:08.569377 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:08.577984 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:08.631668 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:08.685597 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:08.692230 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:08.693140 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:08.694574 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:08.713665 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:06:08.725225 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:08.729445 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:08.730144 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:08.731765 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:08.732549 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:08.740687 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:08.743387 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 08:06:08.751808 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 08:06:08.781383 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:06:08.843057 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:08.847216 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:08.848322 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:08.849107 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:08.852725 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:08.948258 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:08.949136 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:08.951170 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:08.954171 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:08.967039 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:09.047423 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:09.053507 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:09.057420 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:09.063824 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:09.079144 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:09.092075 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240501 08:06:09.095665 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:09.102511 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:09.104386 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:09.105114 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:09.126526 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:09.128167 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:09.176770 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:09.292912 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:06:09.371819 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:06:09.414691 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:09.415005 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 08:06:09.418154 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:09.421783 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:09.423385 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:09.449656 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:06:09.452015 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:09.474751 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:09.475934 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:09.481154 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:09.481154 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:09.498262 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:09.577790 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:09.578006 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:09.581272 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:09.582727 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:09.584331 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:09.617839 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:09.680459 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:06:09.692276 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:09.696211 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:09.699147 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:09.699404 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:09.715449 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:09.789572 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:09.790140 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:09.794435 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:09.818408 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:09.821379 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:09.855746 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:09.856300 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:09.859463 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:09.861053 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:09.871263 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:09.964453 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:09.965502 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:09.967535 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:09.967649 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:09.977770 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:10.117342 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:10.117846 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:10.121687 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:10.126600 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:10.127115 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:10.134013 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:10.251610 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:10.255537 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:06:10.267300 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:10.269616 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:10.270782 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:10.272020 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:10.290158 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:10.371357 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:10.372238 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:10.374125 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:10.378217 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:10.380832 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:10.453598 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:10.459298 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:10.468477 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:10.471365 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:10.488397 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:10.533492 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:10.533512 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:06:10.533939 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:10.534853 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:10.543982 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:10.551836 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:10.566910 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 08:06:10.607661 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 08:06:10.642113 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:10.644668 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:10.645217 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:10.645975 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:10.646951 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:10.749122 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:10.751909 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:10.752913 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:10.761688 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:10.765174 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:10.772568 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:10.792294 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:06:10.822379 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:10.823760 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:10.828892 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:10.834750 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:10.835191 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:10.864037 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:10.922910 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:06:11.073704 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:11.074337 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:11.085364 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:11.087646 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:11.087755 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:11.110347 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 08:06:11.150934 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240501 08:06:11.151077 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:06:11.169616 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:11.177542 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:11.180490 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:11.180547 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:11.210449 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:11.234201 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:11.235504 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:11.240171 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:11.244696 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:11.247871 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:11.247910 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:11.343341 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:11.344061 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:11.348073 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:11.360072 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:11.363408 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:11.429553 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:06:11.447558 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:11.450546 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:11.453481 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:11.454699 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:11.469130 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:11.547938 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:11.549305 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:11.552513 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:11.557938 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:11.573680 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:11.603783 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:11.605367 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:11.611245 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240501 08:06:11.616534 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:11.616631 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:11.626595 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:11.677688 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:11.748942 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:11.809247 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:11.879858 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:11.887176 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:11.887332 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:11.887710 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 08:06:11.890688 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:11.894625 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:11.954829 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:06:11.961968 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:11.963680 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:11.965023 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:11.966607 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:11.967599 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:12.069172 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:12.069730 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:12.071182 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:12.072229 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:12.085644 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:12.183859 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:12.184875 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:12.187180 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:12.188151 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:12.196449 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:12.203241 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:12.229378 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:12.255681 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:12.261399 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:12.264276 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:12.264406 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:12.274003 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:12.336891 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:12.343266 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:12.344578 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:12.349067 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:12.349128 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:12.453119 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:12.455651 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:12.457474 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:12.458742 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:12.475446 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:12.537756 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 08:06:12.575110 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:12.576789 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:12.578298 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:12.579077 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:12.588649 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:06:12.588908 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:12.753860 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:12.766904 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:12.768501 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:12.769196 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:12.780819 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:12.786849 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:12.887028 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:12.892509 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:12.897373 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:06:12.897826 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:12.897912 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:12.900045 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:12.929548 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:12.932027 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:06:12.937379 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:12.940953 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:12.941973 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:12.950034 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:13.043494 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:13.044718 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:13.045248 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:13.046146 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:13.048717 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:13.142484 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:13.144047 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:13.144265 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:13.154461 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:13.159251 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:13.200294 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:13.263887 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:06:13.269410 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:13.272632 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:13.273049 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:13.274030 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:13.292982 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:06:13.299700 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:13.301031 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:13.307193 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:13.313097 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:13.314498 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:13.319491 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:06:13.332051 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:13.423466 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240501 08:06:13.610630 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:13.611896 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:13.615856 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:13.627791 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:13.636404 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:06:13.637877 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:13.665179 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:13.665488 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:13.665975 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:13.671056 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:13.681017 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:06:13.693853 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:13.775048 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:13.776863 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:13.777345 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:13.787869 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:13.794788 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:13.803843 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:13.843787 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 08:06:13.877816 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:06:13.898089 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:13.900355 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:13.904029 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:13.912914 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:13.916306 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:14.022855 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:14.024307 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:14.025085 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:14.030798 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:14.032301 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:14.040586 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:14.043330 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:14.063845 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:14.069481 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:14.071873 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:14.165254 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:14.170245 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:14.175568 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:14.176077 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:14.187686 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:14.239732 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 08:06:14.268494 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:14.308197 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 08:06:14.310467 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:14.355624 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 08:06:14.356035 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:14.358160 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:14.359198 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:14.360759 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:14.360852 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:14.409529 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:06:14.488807 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:14.491080 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:14.492344 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:14.492383 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:14.505890 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:14.519637 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:06:14.592621 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:14.593670 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:14.594228 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:14.596690 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:14.597316 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:14.689802 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:14.694015 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:14.695780 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:14.696298 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:14.706463 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:14.740587 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:14.756297 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:14.758713 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:14.764040 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:14.765559 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:14.769409 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:14.857378 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:14.861936 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:14.862257 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:14.862731 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:14.863566 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:14.930877 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:06:14.979580 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:14.980387 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:14.981201 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:14.983603 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:14.998566 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:15.044873 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:15.049477 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:15.049882 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:15.049932 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:15.058099 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:15.269678 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:06:15.295132 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:15.296329 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:15.297497 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:15.297714 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:15.301796 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:15.304250 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:15.393038 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:15.400014 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:06:15.400635 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:15.406597 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:15.407522 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:15.434140 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:15.443739 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:15.446988 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:15.449388 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:15.450695 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:15.460829 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:15.534554 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:06:15.554467 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:15.555694 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:15.556152 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:15.558455 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:15.562610 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:15.600255 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 08:06:15.600838 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:15.642270 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 08:06:15.644376 11114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 08:06:15.646526 11114 catalog_manager.cc:4573] TS 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c95c4f3ef08f44d1b69fb7b98426f5af: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:06:15.658967 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:15.660306 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:15.661283 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:15.666600 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:15.675580 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:15.775151 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:15.783143 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:15.785174 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:15.801713 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:15.811144 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:15.811591 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:15.827340 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:15.832325 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:15.838970 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:15.847338 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:15.853261 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:15.879495 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:15.930840 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 08:06:16.102000 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:06:16.145043 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:16.151087 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:16.155115 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:16.174367 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:16.177393 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:16.177816 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:16.180315 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:16.180387 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:16.199251 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:16.200527 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:16.220021 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:16.280717 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:16.298830 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:16.301095 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:16.302335 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:06:16.303409 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:16.303563 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:16.304744 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:16.407775 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:16.409225 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:16.419166 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:16.423189 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:16.431253 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:16.522326 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:16.525120 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:16.533730 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:16.537698 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:16.541247 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:16.549595 23443 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:06:16.549736 23443 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:06:16.550159 23443 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
W20240501 08:06:16.555229 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:16.555402 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:06:16.555445 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:06:16.555619 23443 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:06:16.560952 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:16.563025 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:16.563724 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:16.570710 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:16.581257 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 08:06:16.589448 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:16.589783 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:16.664299 11115 catalog_manager.cc:4573] TS e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 18b146cc4e164e1d9f9f827217c9b3ab: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:06:16.678323 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:16.678721 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:16.678925 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:16.683751 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:16.685408 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:16.702450 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:16.840958 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:06:16.854084 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:16.854108 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:16.861344 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:16.862841 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:16.864701 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:16.887792 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 08:06:16.889309 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240501 08:06:16.982383 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:06:16.989094 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:16.990804 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:16.991777 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:17.001250 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:17.015726 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:17.096465 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:17.099195 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:17.099989 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:17.100615 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:17.110610 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:17.191054 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:17.192313 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:17.193688 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:17.195297 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:17.210383 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:17.264838 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:17.265601 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:17.267838 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:17.268474 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:17.269021 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:17.269835 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:17.373355 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:17.374251 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:17.374298 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:17.377017 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:17.378151 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:17.491673 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:17.494555 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:17.496464 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:17.496855 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:17.507401 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:17.558192 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:06:17.558532 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:17.558815 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:17.559640 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:17.573098 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:17.574587 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:17.792028 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:17.824571 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:17.825004 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:17.828390 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:17.837741 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:17.838860 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:17.900075 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:17.900768 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:17.902302 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:17.910681 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:17.920008 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:17.927636 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:17.939644 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:17.966961 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 08:06:17.982913 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:17.983619 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:17.985087 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:17.985421 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:17.985528 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:18.009254 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:06:18.088232 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:18.089819 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:18.092468 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:18.095484 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:18.103353 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:18.196421 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:18.198081 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:18.201571 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:18.201927 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:18.202587 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:18.234154 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:06:18.255720 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:18.256593 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:18.263882 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:18.275071 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:18.277161 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:18.343772 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:18.362576 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:06:18.363605 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:18.363607 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:18.363977 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:18.364275 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:18.378511 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:18.385663 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 08:06:18.452078 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 08:06:18.523948 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:06:18.570632 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:18.570914 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:18.574139 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:18.582198 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:18.594847 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:18.606293 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:06:18.673928 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:18.674225 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:18.677049 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:18.680171 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:18.681706 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:18.776741 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:18.777369 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:18.779843 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:18.781296 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:18.786502 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:18.856196 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:06:18.893159 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:18.899448 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:18.902047 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:18.903157 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:18.905395 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:18.919643 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:18.942504 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:18.950083 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:18.952438 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:18.953434 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:18.956113 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:19.012676 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 08:06:19.027498 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:06:19.040132 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:19.044807 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:19.045233 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:19.045277 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:19.046085 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:19.090261 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:06:19.145905 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:19.146857 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:19.147043 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:19.147433 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:19.151219 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:19.262514 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:19.265265 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:19.265285 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:19.267634 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:19.287955 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:19.323637 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:06:19.427767 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:19.437661 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240501 08:06:19.448583 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:19.452117 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:19.456863 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:19.458640 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:19.462779 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:06:19.464047 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:19.555426 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:19.556946 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:19.564293 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:19.568415 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:19.568411 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:19.616173 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:19.626281 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:19.626910 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:19.628474 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:19.639781 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:19.689927 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 08:06:19.721618 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:19.721709 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:19.728693 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:19.733621 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:19.736889 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:19.831642 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:19.831799 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:19.833423 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:19.833822 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:19.835548 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:19.958020 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:19.961844 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:19.962615 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:19.969676 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:19.971028 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:19.971359 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:19.974861 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:19.975523 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:19.986968 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:19.988201 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:19.990962 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:20.115051 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:20.281059 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:06:20.314209 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:20.320478 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:06:20.321844 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:20.322921 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:20.326218 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:20.333205 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:20.344888 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:20.356920 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:20.359867 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:20.376967 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:20.386660 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:20.391093 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:20.464721 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:20.465143 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:20.467728 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:20.469645 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:20.479967 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:20.532882 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:20.564237 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:06:20.572988 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:20.574149 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:20.578109 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:20.579584 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:20.588285 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:20.648895 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:06:20.652184 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 08:06:20.686769 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:20.692011 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:20.692036 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:20.712087 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:20.712873 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:20.741916 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:20.745568 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:20.749205 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:20.749783 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:20.757067 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:20.851176 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:20.851310 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:20.851954 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:20.852027 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:20.854583 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:20.929231 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:20.946370 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:20.990341 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240501 08:06:21.059046 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:21.062628 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:21.063701 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:21.063913 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:21.070664 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:21.090893 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:06:21.169423 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:21.174810 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:21.174939 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:21.175168 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:21.176040 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:21.270887 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:21.271808 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:21.272667 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:06:21.282514 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:21.285419 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:21.285756 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:21.299253 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:06:21.388037 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:21.389868 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:21.394572 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:21.399168 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:21.401943 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:21.411338 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:06:21.428081 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:21.433919 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:21.437018 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:21.440466 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:21.441442 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:21.451664 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 08:06:21.495433 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:21.516059 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:21.545637 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:21.545680 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:21.548826 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:21.549624 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:21.551643 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:21.658157 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:21.659884 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:21.664604 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:21.664861 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:21.680816 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:21.749974 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:21.754230 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:21.759943 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:21.771142 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:21.782639 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:21.831476 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 08:06:21.901095 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:21.909759 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:06:21.975872 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:21.979300 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:21.982141 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:21.985086 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:21.986789 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:22.033938 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 08:06:22.099154 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:22.101681 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:22.103587 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:22.106621 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:22.106629 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:22.110353 26284 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:06:22.110515 26284 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:06:22.113217 26284 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
W20240501 08:06:22.114449 13122 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:22.116361 13121 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:06:22.116447 13121 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9; no voters: 60e8857abe7c41038af46ebb5a095a90, ca177138f2c44d2eafa1646b795f7608
I20240501 08:06:22.116623 26284 raft_consensus.cc:2726] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:06:22.120828 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:22.122083 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:22.138271 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:22.139712 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:22.148480 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:22.249861 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:22.253615 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:22.254098 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:22.257508 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:22.266048 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:22.350525 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:22.352056 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:22.353404 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:22.357332 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:22.358443 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:22.385278 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:22.464514 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:22.465116 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:22.478610 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:22.480067 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:22.481123 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:22.490077 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:22.500095 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:22.510123 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:22.514834 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:22.528179 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:22.733008 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:22.747288 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:22.773443 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:22.828099 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:22.831601 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:22.835330 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:22.849581 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:22.850100 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:22.880323 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:22.885231 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:22.886480 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:22.899363 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:22.901404 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:22.913795 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:06:22.983237 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:22.984967 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:22.986210 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:06:22.987542 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:22.988158 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:22.989686 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:23.054558 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 08:06:23.089673 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:23.093734 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:23.094467 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:23.099556 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:23.115141 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:23.177215 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:23.181692 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:23.185520 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:23.186937 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:23.193212 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:23.239318 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:23.239599 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:23.253069 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:23.256806 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:23.259102 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:23.287752 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:23.293756 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 08:06:23.317315 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:23.354344 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:23.363025 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:23.367190 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:23.371619 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:23.372494 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:23.520205 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:23.522114 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:23.522765 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:23.523077 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:23.531575 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:23.541072 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:23.563263 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:06:23.638289 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 08:06:23.640076 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:06:23.680776 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:23.682965 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:23.683023 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:23.688122 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:23.690789 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:23.803578 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:23.814271 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:23.815200 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:23.815403 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:23.827692 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:23.879577 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:23.896876 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:23.900282 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:23.900310 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:23.903842 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:23.912199 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:06:23.935725 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:23.989341 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:23.990881 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:23.992725 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:23.993165 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:23.996559 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:24.007757 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:24.091692 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:06:24.093152 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:24.096310 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:24.096328 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:24.099121 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:24.099575 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:24.131608 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:06:24.216015 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:24.216545 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:24.217903 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:24.219154 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:24.234674 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:24.280035 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:24.280992 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:24.285096 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:24.285934 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:24.298179 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:24.388599 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:24.389022 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:06:24.389667 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240501 08:06:24.537856 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:24.545310 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:24.550181 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:24.557046 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:24.557149 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:24.620582 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:24.623699 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:24.623699 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:24.627440 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:24.629026 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 08:06:24.650342 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:24.651084 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:06:24.709704 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:24.710100 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:24.712991 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:24.714802 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:24.716221 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:24.825052 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:24.825799 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:24.829879 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:24.831111 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:24.842592 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:24.935575 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:24.940838 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:24.940840 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:24.941710 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:24.946692 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:24.968678 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:25.002266 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:25.007414 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:25.007701 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:25.027211 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:25.031528 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:25.090016 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:25.092439 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:25.092777 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:25.095383 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:25.104063 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:25.323618 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:25.325654 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:25.328667 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:25.338555 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:25.347743 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:25.422065 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:25.423468 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:25.425355 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:25.427829 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:25.434772 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:25.517140 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:25.523243 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:06:25.524196 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:25.528738 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:25.529609 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:25.534610 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:25.538645 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:25.547374 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:25.602299 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:25.639942 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:06:25.651484 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:25.653019 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:25.653235 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 08:06:25.660868 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 08:06:25.668074 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:25.668962 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:25.669715 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:25.674569 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:25.674743 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:25.682971 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:25.687585 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:25.695441 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 08:06:25.701941 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:06:25.710834 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:25.795815 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:25.795847 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:25.797757 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:25.797875 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:25.820842 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:25.909058 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:25.909155 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:25.910346 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:25.911252 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:25.912621 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:26.007462 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:26.008028 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:26.011090 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:26.011775 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:26.012135 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:26.020820 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:26.030514 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 08:06:26.094240 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:06:26.236480 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:26.237573 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:26.238368 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:26.239238 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:26.249502 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:26.320717 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:06:26.339145 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:26.341722 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:26.345742 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:26.366650 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:26.378445 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:26.383350 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:26.386354 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:26.398044 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:26.401638 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:26.405205 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:26.469563 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:26.494217 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:06:26.501034 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:26.501920 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:26.503150 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:26.505069 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:26.505736 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:26.546553 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:06:26.607759 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:26.609624 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:26.611418 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:26.612843 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:26.625051 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:26.652375 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:06:26.728806 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:26.731575 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:06:26.742697 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:26.746553 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:26.756182 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:26.759547 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:26.760584 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:26.767947 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:26.768505 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:26.770552 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:26.791508 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:26.974789 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240501 08:06:27.018990 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:27.081918 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 08:06:27.111268 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:27.112270 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:27.114261 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:27.119436 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:27.119724 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:06:27.122370 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:27.122717 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:27.123554 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:27.124826 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:27.155593 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:27.168889 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:27.251042 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:27.253003 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:27.254040 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:27.264307 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:27.272083 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:27.367599 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:27.368036 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:27.368539 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:27.368553 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:27.369992 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:27.485725 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:27.488366 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:27.496783 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:27.498392 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:27.498796 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:27.502542 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:27.522359 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:27.524163 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:27.524994 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:27.541182 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:27.627349 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:27.649622 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:27.653523 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:27.653601 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:27.654148 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:27.668879 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:27.829493 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:27.830041 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:27.830615 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:27.835330 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:27.844645 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:27.945900 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:06:27.963801 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:27.966778 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:27.966823 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:27.968870 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:27.981176 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:28.065168 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:28.068069 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:28.068266 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:28.069283 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:28.075223 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:28.076561 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:28.095945 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:28.136790 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:28.161345 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:06:28.167204 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:28.167778 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:28.168865 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:28.172201 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:28.193271 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:28.203784 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:06:28.235985 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:28.236063 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:28.237402 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:28.239881 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:28.246402 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:28.333338 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:28.336275 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:28.337241 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:28.343286 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:28.345125 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:28.382066 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:06:28.453186 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:28.455060 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:28.456234 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:28.457095 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 08:06:28.466403 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:28.470140 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:28.526005 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:28.527386 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:28.529516 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:28.533006 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:28.553661 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:28.561182 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:28.581764 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 08:06:28.632304 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 08:06:28.670097 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:06:28.768366 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:28.769393 26284 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:06:28.769502 26284 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:06:28.769955 26284 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
W20240501 08:06:28.771533 13121 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:06:28.772540 13122 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:28.772621 13122 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9; no voters: 60e8857abe7c41038af46ebb5a095a90, ca177138f2c44d2eafa1646b795f7608
I20240501 08:06:28.773178 26284 raft_consensus.cc:2726] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:06:28.774710 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:28.776031 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:28.776124 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:28.776463 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:28.807559 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:28.821980 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:06:28.869738 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:28.874822 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:28.880355 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:28.888576 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:28.890733 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:28.913287 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:28.916888 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:28.918525 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:28.925210 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:28.926241 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:28.961016 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:29.042368 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:29.043792 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:29.048787 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:06:29.049459 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:29.057349 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:29.060276 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:29.080144 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:06:29.140480 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:29.144542 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:29.147991 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:29.154649 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:29.155555 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:29.259617 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:29.266861 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 08:06:29.270089 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:29.273380 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:29.285287 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:29.292999 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:29.295359 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:29.297792 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:29.300956 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:29.323050 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:29.326542 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:29.436197 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 08:06:29.441638 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:29.451776 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 08:06:29.536381 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 08:06:29.538257 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:06:29.638568 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:29.639626 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:29.641589 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:29.642143 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:29.647374 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:29.656630 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:29.657466 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:29.663285 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:29.666904 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:29.707588 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:29.792982 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:29.799069 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:29.799665 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:29.800824 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:29.807515 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:29.914050 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:29.914075 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:29.914690 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:29.923027 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:29.946657 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:29.978932 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:30.019469 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:30.019768 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:30.021994 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:30.022809 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:30.029218 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:30.085486 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:30.085826 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:30.086990 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:30.088626 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:30.111065 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:30.194294 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:30.198853 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:30.199486 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:30.199750 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:30.200546 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:30.331115 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:30.334672 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:30.336745 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:30.343545 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:30.345638 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:30.456935 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:30.499279 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:06:30.512082 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:30.514408 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:30.514631 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:30.523540 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:30.525796 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:30.576151 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:30.616981 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:06:30.624490 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:30.625537 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:30.628372 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:30.632949 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:06:30.636772 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:30.638777 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:30.693949 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:30.697000 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:30.697921 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:30.706725 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:30.718330 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:30.762048 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:06:30.768234 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:30.769294 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:30.769575 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:30.776196 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:30.778756 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:30.830523 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:06:30.868688 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:30.870234 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:30.872959 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:30.877835 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:30.879477 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:30.899158 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:06:30.949077 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 08:06:30.987609 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:30.988008 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:30.995635 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:30.996981 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:31.014992 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:31.026883 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:31.056190 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:31.056679 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:31.057003 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:31.057546 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:31.059806 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:31.122879 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:31.170296 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:06:31.314491 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:31.317612 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:31.322829 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:31.322921 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:31.326272 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:31.393508 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:31.399435 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:31.399672 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:31.411412 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:31.424041 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:31.464927 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 08:06:31.467339 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240501 08:06:31.472398 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:31.472697 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:31.475462 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:31.477865 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:31.479205 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:31.539980 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:06:31.580485 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:31.581295 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:31.584861 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:31.586745 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:31.597355 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:31.620929 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:31.700074 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:31.705247 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:31.707068 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:31.707146 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:31.716167 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:31.766497 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:06:31.769884 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:31.771839 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:31.777079 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:31.777199 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:31.777695 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:31.853674 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:06:31.855145 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:31.855549 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:31.859874 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:31.862560 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:31.867056 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:31.894775 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:31.930527 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 08:06:32.023789 26284 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:06:32.023957 26284 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:06:32.024437 26284 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
W20240501 08:06:32.025471 13121 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:06:32.025560 13122 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:32.025619 13122 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9; no voters: 60e8857abe7c41038af46ebb5a095a90, ca177138f2c44d2eafa1646b795f7608
I20240501 08:06:32.025758 26284 raft_consensus.cc:2726] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:06:32.089821 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:32.091953 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:32.094184 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:32.095789 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:32.115193 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:32.181299 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20240501 08:06:32.188182 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:32.189540 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:32.190352 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:32.193018 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:32.206032 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:32.220620 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:32.290875 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:32.292663 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:32.293780 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:32.294368 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:32.297606 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:32.407660 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:32.411446 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:32.415145 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:32.419971 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:32.434834 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:32.450862 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:32.452075 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:32.466315 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:32.468142 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:32.478080 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:32.555928 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:32.556316 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:32.558179 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:32.558498 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:32.559801 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:32.662503 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:32.663300 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:32.665874 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:32.666958 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:32.670646 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:32.722239 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:32.775432 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:32.776154 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:32.777360 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:32.778870 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:32.781559 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:32.933329 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:06:32.971307 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:32.972295 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:32.974710 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:32.975417 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:32.976166 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:33.083657 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:33.092520 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:33.094511 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:06:33.107522 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:33.111598 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:33.111685 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:33.113081 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:33.114885 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:33.122853 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:33.145272 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:33.147331 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:06:33.156297 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:33.234042 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:33.237470 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:33.239168 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:33.239526 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:33.243458 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:33.271204 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:33.330789 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:33.331483 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:33.332237 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:33.342037 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:33.346016 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:33.359777 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:33.363786 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:33.411092 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:33.465227 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:06:33.466873 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:33.467455 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:33.476816 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:33.477290 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:33.478607 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:33.489730 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:33.497486 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:33.512627 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:33.517176 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:33.523516 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:33.605417 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:06:33.627370 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:33.809233 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:06:33.809934 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:33.820901 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:33.821979 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:33.831125 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:33.833235 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:33.863732 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 14 similar messages]
I20240501 08:06:33.876384 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:33.877440 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:33.878885 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:33.881044 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:33.882685 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:33.889528 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:33.972046 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:33.973124 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240501 08:06:33.975304 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:33.978540 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:33.978646 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:33.980258 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:34.036921 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:34.077304 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:06:34.082136 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:34.085242 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:34.087836 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:34.094832 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:34.103662 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:34.173943 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:34.179656 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:34.180207 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:34.180459 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:34.181351 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:34.223727 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 08:06:34.240494 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:34.243479 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:34.244339 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:34.244939 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:06:34.251502 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:34.258121 30446 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:06:34.258296 30446 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:06:34.258769 30446 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
W20240501 08:06:34.260020 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:34.260126 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:06:34.260200 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:06:34.260426 30446 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:06:34.262984 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:34.283109 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 08:06:34.322304 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:34.353371 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:34.354938 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:34.355054 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:34.357400 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:34.366824 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:34.436363 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 08:06:34.498947 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:34.499773 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:34.503193 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:34.505759 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:34.507160 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:34.664443 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:34.664436 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:34.665303 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:34.667424 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:34.669430 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:34.780033 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:34.780782 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:34.783290 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:34.783553 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:34.812234 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:34.849839 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:34.851763 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:34.857467 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:34.858371 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:34.860913 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:34.880745 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 08:06:34.895337 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:34.954223 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:34.955904 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:34.956425 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:34.956885 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:34.959167 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:35.053038 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:35.055718 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:35.058593 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:35.058805 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:35.067162 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:35.171262 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:35.172770 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:35.177937 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:35.191082 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:35.200101 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:35.223517 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:35.233299 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:35.233299 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:35.241433 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:35.248581 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:35.395310 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:35.433073 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:35.521440 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:35.524960 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:35.526293 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:35.526556 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:35.541878 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:35.587288 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:35.591192 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:35.591882 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:35.599049 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:35.604054 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:35.611518 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:06:35.669157 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:35.670480 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:35.671676 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:35.673117 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:35.674533 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:35.760071 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:06:35.783134 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:35.783742 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:35.784181 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:35.784350 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:35.784518 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:35.795102 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:35.818948 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:35.874361 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:35.876492 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:06:35.888530 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:35.892825 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:35.897325 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:35.897609 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:35.904139 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:35.931761 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 08:06:35.947085 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:35.956203 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:35.956817 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:35.958927 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 08:06:35.959137 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:06:35.960712 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:35.970064 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:36.058494 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:36.059857 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:36.061923 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:36.070628 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:36.072149 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:36.181666 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 08:06:36.269335 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:36.271749 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:36.274237 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:36.282200 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:36.291167 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:36.352074 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:36.371475 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:36.373685 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:36.376758 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:36.384697 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:36.392683 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:36.427999 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 08:06:36.429797 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:06:36.482455 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:36.482494 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:36.482874 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:36.484091 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:36.493672 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:36.594830 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:36.595548 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:36.598066 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:36.616151 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:36.616638 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:06:36.619448 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:36.624568 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:36.634521 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:36.643227 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:36.665195 11249 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:36.665405 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:36.684768 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:06:36.747236 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:36.749186 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:36.750144 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:36.759277 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:06:36.760171 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:36.766222 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:36.772969 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:36.804905 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 08:06:36.860908 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:36.862314 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:36.864212 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:06:36.869001 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:36.871595 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:36.890983 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:36.965025 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:36.967362 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:36.969025 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:36.977802 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:36.982471 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:37.017720 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240501 08:06:37.180358 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:37.181504 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:37.181859 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:37.182389 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:37.185242 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:37.254184 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:37.289029 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:37.290153 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:37.291776 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:37.303128 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:37.307644 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:37.336552 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:37.341863 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:37.342185 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:37.342682 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:37.342800 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:37.441795 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:37.448145 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:37.449821 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:37.450572 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:37.453730 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:37.542927 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:37.549125 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:37.549714 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:37.552093 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:37.557415 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:37.642072 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:37.649387 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:37.649437 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:37.653234 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:37.680868 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:37.692400 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:37.692394 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:37.695535 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:37.696818 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:37.709563 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:37.713608 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:37.855741 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:06:37.987920 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:37.988770 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:37.989346 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:37.989590 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:37.999081 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:38.037688 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:06:38.050930 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:38.053041 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:38.057360 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:38.058669 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:38.067459 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:38.083298 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:38.113924 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:06:38.152881 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:38.154278 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:38.154376 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:38.163432 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:38.171712 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:38.264547 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:38.264891 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:38.271234 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:38.271504 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:38.279125 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:38.282923 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:38.333544 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:06:38.350683 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:38.352423 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:38.362175 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:38.362455 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:38.367076 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:38.409046 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:06:38.409988 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:38.414086 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:38.414505 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:06:38.414736 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:38.427515 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:38.440142 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:38.527791 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:38.529080 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:38.529585 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:38.540454 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:38.546305 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:38.578439 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 08:06:38.677012 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:38.690187 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:38.692596 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:38.692767 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:38.699908 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:38.703428 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:38.737284 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:06:38.840601 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:38.841301 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:38.843036 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240501 08:06:38.843418 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:38.846724 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:38.862053 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:38.943004 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:06:38.945611 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:38.953923 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:38.954386 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:06:38.957049 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:38.958642 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:38.962615 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:39.038265 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:39.039547 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:39.043233 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:39.046506 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:39.063616 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:39.099294 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:39.100298 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:39.106343 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:39.114284 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:39.126892 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:39.208173 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:39.212109 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:39.213397 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:39.223888 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:39.227819 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:06:39.235116 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:39.245325 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:39.332762 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:39.332880 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 08:06:39.333252 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:39.339406 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:39.343848 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:39.346413 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:39.382579 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:06:39.388419 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:39.396421 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:39.398782 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:39.400880 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:39.418623 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:39.426167 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:06:39.427202 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:39.463127 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:06:39.573367 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240501 08:06:39.658902 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:39.662575 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:39.664083 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:39.672641 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:39.673331 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:39.748629 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:39.754619 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:39.755177 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:39.755722 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:39.761219 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:39.815073 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:39.820529 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:39.820817 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:39.821242 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:39.824007 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:39.834650 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:39.923404 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:39.925923 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:39.930167 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:39.933548 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:39.934592 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:40.034204 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:40.038534 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:40.038901 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:40.048566 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:40.054579 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:40.101508 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:40.101542 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:40.107259 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:40.109459 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:40.117158 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:40.199643 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:40.200850 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:40.203032 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:40.203668 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:40.204813 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:40.215476 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:40.385362 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:40.405174 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:40.405405 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:40.411190 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:40.419198 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:40.428234 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:40.511895 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:40.513378 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:40.514274 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:40.515673 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:40.517282 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:40.607388 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:40.607896 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:40.611553 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:40.614048 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:40.620222 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:40.627614 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:40.701395 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:06:40.730342 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:40.737098 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:40.739413 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:40.740820 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:40.743451 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:40.747560 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:40.752887 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:40.752997 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:06:40.769611 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:40.771493 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:40.794857 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:40.842082 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:06:40.881400 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:40.883544 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:40.883781 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:40.885700 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:40.893062 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:40.923897 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:40.929256 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:40.931365 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:06:40.984233 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:40.986148 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:40.987192 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:40.988510 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:40.998391 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:41.083544 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:41.084038 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:41.085646 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:41.095446 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:41.095749 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:41.097779 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 08:06:41.206080 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 08:06:41.219993 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240501 08:06:41.289588 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:41.290530 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:41.290863 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:41.293727 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:41.297567 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:41.310294 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 08:06:41.322436 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:41.357695 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:06:41.405952 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:41.408144 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:41.411862 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:41.414407 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:41.420413 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:41.421471 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:41.427569 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:41.433271 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:41.450805 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:41.473023 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:41.533255 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:41.553735 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:41.557066 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:41.558591 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:41.558804 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:41.559851 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:41.652792 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:41.654115 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:41.656344 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:41.661595 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:41.663280 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:41.736737 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:06:41.784950 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:41.785841 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:41.787401 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:06:41.788096 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:41.788506 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:41.794224 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:41.799682 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 08:06:41.807792 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:41.809469 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:41.809604 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:41.826508 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:41.826961 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:41.860430 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:41.927719 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:42.147042 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:42.147254 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:42.148267 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:42.153430 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:42.158128 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:42.159957 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:42.169502 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:42.171936 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:42.181123 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:42.195353 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:42.282464 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240501 08:06:42.287452 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:42.290325 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:42.295121 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:42.299814 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:42.310597 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:42.393098 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:42.396703 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:42.396976 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:42.397259 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:42.400671 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:42.511204 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:42.511936 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:42.512470 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:42.521687 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:42.523476 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:42.535266 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:42.540165 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:42.547539 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:42.552621 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:42.556691 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:42.575840 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:42.613143 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:06:42.654474 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:42.656024 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:42.658689 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:42.662581 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:42.671300 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:42.865694 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:42.866210 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:42.867027 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:42.875310 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:42.888526 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:42.929791 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:06:42.964129 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:42.965243 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:42.965801 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:42.967957 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:42.977319 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:43.061601 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:43.064757 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:43.066582 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:43.069808 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:43.079332 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:43.079473 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:43.179514 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:43.186170 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:43.186265 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:43.187117 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:43.191593 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:43.194195 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:06:43.198693 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:43.217984 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:43.222095 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:43.228034 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:06:43.235634 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:43.237010 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:43.296797 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:06:43.320298 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:43.322305 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:43.322721 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:43.322841 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:43.334584 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:43.417130 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:43.426839 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:06:43.434084 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:43.434214 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:43.438653 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:43.448137 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:06:43.450129 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:43.451440 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:43.470819 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:43.539091 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:43.539301 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:43.539515 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:43.539713 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:43.540320 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:43.743297 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:43.744357 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:43.745810 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:43.755352 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:43.757444 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:43.767091 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:43.818138 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:06:43.866324 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:43.869930 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:43.874831 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:43.874867 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:43.877125 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:43.878698 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:43.888345 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:43.899866 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:43.907642 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:43.909757 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
I20240501 08:06:43.912377 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:43.935981 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:44.016024 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:44.020867 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:44.024752 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:44.027487 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:44.030611 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:44.085232 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 08:06:44.088459 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:06:44.119287 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:44.120859 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:44.122228 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:44.123888 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:44.128675 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:44.213747 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 08:06:44.239450 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:44.242108 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:44.256889 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:44.259704 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:44.264891 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:44.268734 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:44.275960 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:44.281383 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:44.281527 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:44.295886 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:44.347646 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:44.415030 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:44.441608 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:44.555333 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 08:06:44.594005 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:44.596828 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:44.599718 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:44.619367 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:44.620424 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:44.641031 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:44.652115 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:44.652493 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:44.657290 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:44.657668 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:44.754915 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:44.760854 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:44.763986 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:44.766347 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:44.772719 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:44.849936 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 08:06:44.863713 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:44.864522 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:44.866240 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:44.870616 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:44.873510 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:44.887064 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240501 08:06:44.965714 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:44.968133 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:44.968201 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:44.980600 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:44.983631 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:44.991686 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:45.027726 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:45.030766 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:45.032653 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:45.039582 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:45.040170 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:45.068338 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:06:45.132521 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:45.133167 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:45.134830 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:45.136752 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:45.146440 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:45.284849 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:45.284907 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:45.287223 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:45.289570 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:45.295068 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:45.356796 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:06:45.439862 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:45.445209 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:45.446772 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:45.447222 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:45.449340 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:45.514242 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:06:45.543526 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:45.547623 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:45.548345 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:45.548393 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:45.554715 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:45.557256 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:45.598775 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:06:45.623914 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:45.626343 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:45.629199 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:45.646366 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:45.656826 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:45.706943 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:45.708534 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:45.712303 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:45.719990 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:45.720347 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:45.808912 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:45.815531 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:45.819396 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:06:45.822934 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:45.831681 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:45.836921 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:45.844602 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:06:45.926455 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:45.927560 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:45.937613 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:45.938086 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:45.940487 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:45.955168 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 08:06:45.984795 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:06:45.995754 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:45.995754 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:46.001519 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:46.001487 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:46.005129 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:46.034610 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:46.047701 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 08:06:46.064795 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:46.238559 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 08:06:46.253494 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:46.257615 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:46.258445 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:46.260736 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:46.262090 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:46.323541 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:46.329227 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:46.330121 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:46.332477 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:46.334493 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:46.396924 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:46.397225 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:46.398496 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:46.404181 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:46.407872 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:46.416630 3704 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:06:46.416749 3704 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:06:46.417111 3704 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
W20240501 08:06:46.418795 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:46.419147 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:06:46.419200 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:06:46.419469 3704 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:06:46.433120 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 08:06:46.453495 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:06:46.507728 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:46.507897 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:46.510592 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:46.513238 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:46.520301 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:46.520386 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:46.537158 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:06:46.613222 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:46.613277 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:46.616992 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:46.626829 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:46.631826 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:46.663537 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 08:06:46.688462 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:46.689934 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:46.696527 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:46.703940 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:46.718232 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:46.760031 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:46.760234 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:46.769901 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:46.770222 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:46.776186 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:46.853783 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:46.935799 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:46.987790 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:47.007774 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:47.012135 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:47.013108 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:47.018107 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:47.018561 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:47.096432 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:47.098873 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:47.102008 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:47.109489 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:47.119074 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:47.201081 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:47.201622 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:47.204857 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:47.205866 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:47.212109 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:47.328444 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:47.329772 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:47.331637 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:47.333983 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240501 08:06:47.337728 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:47.342252 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:47.344483 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:47.358650 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:47.364334 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:47.375905 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:47.384568 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:47.461588 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:47.464818 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:47.465725 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:47.467876 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:47.469221 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:47.472558 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:47.491515 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:47.566984 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:47.567971 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:47.571184 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:47.571346 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:47.572172 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:47.672693 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:47.676499 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:47.679040 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:47.679517 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:47.683187 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:47.877991 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:47.879906 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:47.880649 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:47.881477 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:47.882664 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:47.989717 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:47.993474 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:47.996163 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:47.997220 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:48.009178 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:48.011343 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:48.019526 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:48.025398 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:48.028424 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:48.031940 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:48.058874 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:48.066951 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:48.086639 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:06:48.124816 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:48.130549 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:48.132427 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:48.134970 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:48.144032 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:48.179652 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:06:48.225499 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:48.226791 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:48.226938 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:48.228219 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:48.238653 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:48.343367 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:48.350199 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:48.354965 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:48.355345 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240501 08:06:48.366021 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:48.376710 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:48.394290 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:48.394409 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:48.399312 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:48.417811 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:48.426065 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:48.451215 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:48.485797 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:48.500154 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 08:06:48.500720 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:06:48.692699 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:48.696135 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:48.707080 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:06:48.708160 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:48.708432 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:48.725129 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:48.739802 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:06:48.762534 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:48.766381 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:48.766835 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:48.767273 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:48.776078 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:48.800648 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:06:48.838894 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:06:48.862659 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:48.864820 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:48.865018 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:48.865203 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:06:48.866899 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:48.868723 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:48.895445 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:48.969067 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:48.969691 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:48.972718 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:48.978567 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:48.988696 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:49.087414 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:49.089208 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:49.090445 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:49.098834 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:49.102028 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:49.112102 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:49.117587 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:49.123548 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:49.126614 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:49.151152 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:49.186133 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:06:49.225970 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 08:06:49.230078 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:49.232707 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:49.233100 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:49.248322 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:49.250567 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:49.404356 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:49.422104 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:06:49.441077 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:49.443910 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:49.444676 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:49.454818 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:49.456465 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:49.542656 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:49.542642 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:49.549090 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:49.549471 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:49.550931 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:49.559387 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 08:06:49.643373 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:49.645624 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:49.649655 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:49.652451 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:49.662338 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:49.774673 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:49.777921 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:49.781683 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:49.787153 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:49.787451 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:49.792338 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:49.798661 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:49.804208 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:49.821818 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:49.834868 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:49.835886 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240501 08:06:49.913064 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:06:49.916620 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 08:06:49.918648 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:49.919377 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:49.923112 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:49.932945 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:49.946147 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:50.002053 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:50.033258 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:50.034345 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:50.036448 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:50.039155 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:50.039973 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:50.130848 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:50.132552 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:50.133273 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:50.144222 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:50.150030 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:50.338254 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:50.341961 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:50.343075 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:50.343534 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:50.345306 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:50.370345 1523 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:06:50.370494 1523 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:06:50.370970 1523 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
W20240501 08:06:50.372064 13122 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:50.372378 13121 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:06:50.372452 13121 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9; no voters: 60e8857abe7c41038af46ebb5a095a90, ca177138f2c44d2eafa1646b795f7608
I20240501 08:06:50.372653 1523 raft_consensus.cc:2726] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:06:50.454300 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:50.455235 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:50.456022 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:50.458053 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:50.470368 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:50.480805 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:50.489524 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:50.502383 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:50.506625 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:50.507664 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:50.544957 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:06:50.591621 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:50.591915 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:50.592811 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:50.592895 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:50.596170 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:50.602707 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:50.650717 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:06:50.686476 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:50.690367 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:50.691617 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:50.694597 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:50.700604 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:50.797751 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:50.799438 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:50.800262 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:50.800508 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:50.803800 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:50.859594 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240501 08:06:50.864261 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:50.865126 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:50.868449 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:50.870630 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:50.871381 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:51.014272 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:06:51.023792 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:06:51.057581 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:06:51.060464 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:06:51.066054 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:06:51.101500 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 08:06:51.112179 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:51.113020 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:51.114930 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:51.116815 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:51.117623 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:51.123487 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:51.205035 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:51.215541 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:51.216053 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:51.218792 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:51.246865 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:06:51.262027 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:51.264703 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:51.265070 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:51.267969 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:51.268095 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:51.280122 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:51.342965 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:06:51.374454 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:51.374576 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:51.375628 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:51.375946 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:51.377804 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:51.466910 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:51.467049 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:51.480906 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:51.482276 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:51.483647 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:51.492520 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 08:06:51.525703 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:06:51.564076 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:51.574962 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:51.577719 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:51.588272 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:51.592237 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 08:06:51.598870 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:51.599176 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:51.604775 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:51.609175 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:51.622625 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:51.637220 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:51.644942 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:51.763207 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:06:51.932399 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:51.939831 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:51.954262 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:51.967860 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:51.968529 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:51.985210 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:51.989277 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:51.992763 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:52.015424 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:52.019572 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:52.039984 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:06:52.109544 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:52.109961 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:52.112879 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:52.114059 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:52.127627 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:52.166443 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:52.214763 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:52.215575 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:52.216056 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:52.226306 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:52.226852 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:52.275579 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:06:52.322705 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:52.326131 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:52.331681 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:52.337107 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:52.337366 11381 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:52.347662 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:52.359865 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:52.361268 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:52.361830 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:52.369588 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:52.434743 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
I20240501 08:06:52.464198 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:52.466756 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:52.466861 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:52.467126 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:52.467634 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:52.556038 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:52.657143 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:52.658007 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:52.659355 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:52.660442 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:52.672101 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:52.779044 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:52.783167 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:52.784193 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:52.784601 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:52.786674 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:52.888207 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:52.889531 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:52.890867 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:52.893345 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:52.902576 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:52.990173 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:52.990969 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:52.992997 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:52.993705 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:53.005091 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:53.044735 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:53.046705 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:53.050776 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:53.051291 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:53.051381 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:53.094792 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:06:53.098867 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:53.145115 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:53.147197 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:53.148005 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:53.148361 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:53.148944 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:53.197796 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:06:53.251224 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:53.253497 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:53.256485 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:53.263444 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:53.269177 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:53.313979 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:06:53.328635 6563 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:06:53.328789 6563 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:06:53.329193 6563 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
W20240501 08:06:53.330477 13121 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:06:53.330649 13122 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:53.330725 13122 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9; no voters: 60e8857abe7c41038af46ebb5a095a90, ca177138f2c44d2eafa1646b795f7608
I20240501 08:06:53.330880 6563 raft_consensus.cc:2726] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:06:53.342011 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:53.342995 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:53.343410 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:53.350590 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:53.360702 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240501 08:06:53.363715 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:53.366919 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:06:53.374312 4825 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:06:53.374475 4825 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:06:53.374892 4825 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
W20240501 08:06:53.376098 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:53.376219 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:06:53.376266 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:06:53.376395 4825 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:06:53.494366 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:06:53.506182 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:53.525432 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:06:53.542935 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:06:53.566697 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:53.567312 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:53.568694 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:53.573694 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:53.574767 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:53.613276 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:06:53.678611 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:53.685262 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:53.690085 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:53.696540 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:53.704556 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:53.704578 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:53.711678 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:53.725936 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:53.726526 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:53.732899 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:53.834775 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:06:53.841154 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:53.842221 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:53.842303 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:53.846619 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:53.855036 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:53.863153 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 08:06:53.943126 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:53.943686 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:53.947438 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:53.951056 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:53.980036 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:54.020294 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 08:06:54.023844 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:06:54.040346 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:54.043035 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:54.043773 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:54.060807 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:54.068154 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:54.102881 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:06:54.108899 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:54.109040 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:54.114012 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:54.114464 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:54.124166 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:54.125679 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:54.356767 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:54.360545 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:54.361661 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:54.362581 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:54.369987 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:54.423023 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:06:54.456142 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:54.457258 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:54.458117 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:54.460757 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:54.470736 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:54.553231 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:54.554591 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:54.556216 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:54.557343 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:54.557356 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:54.572363 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:54.574391 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 08:06:54.677141 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:54.678151 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:54.684410 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:54.687103 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:54.693208 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:54.696071 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:54.703634 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:54.708624 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:54.716749 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:54.724066 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:54.814808 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:54.815207 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:54.816004 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:54.817689 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:54.836932 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:54.873967 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:06:54.918113 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240501 08:06:54.924983 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:54.926702 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:54.926883 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:54.927376 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:54.936846 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:55.026561 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:55.030292 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:55.032095 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:55.034970 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:55.041625 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20240501 08:06:55.047446 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:55.119614 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:55.236167 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:55.236804 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:55.241685 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:55.243775 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:55.246479 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:55.342542 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:55.343140 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:55.344844 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:55.367590 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:55.369963 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:55.384285 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:55.388592 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:55.388839 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:55.394726 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:55.411072 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:55.500077 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:55.503955 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:55.504494 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:55.505178 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:55.516501 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:55.530104 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:06:55.600185 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:55.600898 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:55.603082 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:55.604138 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:55.606437 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:55.706701 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:55.723343 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:55.725721 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:55.729200 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:55.732323 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:55.732451 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:55.736205 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:55.743077 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:06:55.775014 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:55.775148 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:55.775759 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:55.789773 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:55.797137 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:06:55.866490 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:06:55.945770 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 08:06:56.023234 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240501 08:06:56.044068 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:06:56.086318 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:56.086789 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:56.090646 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:06:56.091495 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:56.101438 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:56.101675 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:56.111131 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:06:56.122470 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:06:56.123234 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:56.124848 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:56.127569 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:56.142731 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:56.147750 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:56.154637 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:56.214234 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:56.225680 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:56.226670 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:56.234668 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:56.234747 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:56.263024 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:06:56.324405 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:56.326138 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:56.327117 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:56.328104 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:56.348256 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:56.428874 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:56.431454 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:56.432691 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:56.452538 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:56.468263 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 08:06:56.468391 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:56.493209 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:56.493991 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:56.494765 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:56.504262 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:56.506142 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:56.541359 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:06:56.558187 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 08:06:56.558880 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:56.584022 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:06:56.593043 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:56.594959 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:56.595657 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:56.596700 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:56.603636 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:56.800208 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:56.800459 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:56.807286 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:56.814353 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:56.818174 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:56.906881 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:56.907208 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:56.911522 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:56.916976 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:56.918946 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:56.950840 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:06:57.012490 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:57.012739 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:57.013319 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:57.016078 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:57.018570 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:57.055279 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:06:57.134970 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:57.138942 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:57.142597 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:57.151446 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:57.154240 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:57.160832 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:57.162149 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:57.166736 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:57.166777 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:57.191421 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:57.201169 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:57.282843 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:57.286614 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:57.287461 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:57.288920 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:57.290402 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:57.344873 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240501 08:06:57.390846 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:57.393688 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:57.394779 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:57.395524 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:57.412027 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:57.499132 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:57.501914 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:57.502400 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:57.503296 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:57.504262 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:57.580808 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:57.708195 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:57.708469 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:57.711372 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:57.713907 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:57.730983 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:57.817698 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:57.821576 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:57.829115 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:57.830689 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:57.833674 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:57.867208 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:57.873983 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:57.876304 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:57.882593 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:57.891570 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:57.977610 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:57.977988 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:57.978973 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:57.979398 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:57.979846 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:58.007879 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:58.070992 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:58.075351 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:58.075955 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:58.077370 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:58.086681 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:58.170560 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:58.171747 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:58.174983 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:58.175921 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:58.184269 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:58.230369 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:58.233080 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:58.234985 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:58.242334 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:58.247898 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:58.330192 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:06:58.375128 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:06:58.400388 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:06:58.420810 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:06:58.493613 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:58.494890 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:58.495915 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:58.502493 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:06:58.502873 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:58.518972 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:58.542762 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:06:58.546895 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:06:58.551316 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240501 08:06:58.569777 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:58.573756 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:58.575493 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:58.575520 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:06:58.577190 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:06:58.584712 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:58.623314 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:58.672796 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:58.673583 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:58.678615 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:58.678706 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:58.692009 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:58.698868 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 08:06:58.722573 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:06:58.768815 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 08:06:58.796447 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:58.799794 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:58.800315 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:58.828132 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:58.831669 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:58.866219 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:58.866873 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:58.874243 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:58.883013 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:58.883471 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:58.961201 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:58.961571 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:58.966791 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:58.967362 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:58.987277 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:59.073063 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:59.076857 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:59.077637 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:59.082459 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:59.095068 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:06:59.177856 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:06:59.179486 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:59.179836 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:59.184834 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:59.186172 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:59.190904 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:59.299428 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:06:59.381295 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:59.382052 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:59.386862 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:59.392647 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:59.394739 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:06:59.404700 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:06:59.473963 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 08:06:59.474730 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:06:59.502655 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:59.502832 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:59.503873 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:59.509544 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:59.519789 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:59.537513 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:59.538810 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:59.551503 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:06:59.551869 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240501 08:06:59.554759 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:59.559350 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:06:59.594369 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:06:59.610587 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240501 08:06:59.645495 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:59.650593 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:59.653602 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:59.653625 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:59.656514 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:59.747078 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:59.751103 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:59.751387 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:59.753882 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:59.773036 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:59.874763 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:59.885591 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:59.886328 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:06:59.890527 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:06:59.891417 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:06:59.891604 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:59.907948 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:06:59.913424 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:59.925659 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:06:59.930207 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:00.117208 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:00.250752 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:00.258363 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:00.261204 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:00.276911 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:00.277292 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:00.278544 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:00.279435 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:00.280392 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:00.282112 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:00.308514 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:00.396977 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:00.397001 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:00.399492 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:00.401291 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:00.419574 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:00.511862 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:00.514351 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:00.516320 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:00.516649 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:00.523981 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:00.542136 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 08:07:00.611802 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:00.616175 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:00.616580 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:00.616856 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:00.625407 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:00.632926 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:00.671136 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:00.671658 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:00.672972 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:00.678921 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:00.686774 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:00.735855 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:07:00.781841 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:00.782125 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:00.786319 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:07:00.788446 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:00.789328 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:07:00.791383 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:00.798584 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:00.944227 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:00.945091 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:07:00.952570 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:00.956763 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:00.957960 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:00.966902 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:00.968820 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:01.005443 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:01.031702 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:01.084309 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 08:07:01.090050 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:07:01.096593 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:01.099017 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:01.102447 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:01.110329 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:01.118671 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:01.171833 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 08:07:01.178030 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:07:01.203099 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:01.204684 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:01.206068 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:01.206732 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:01.211980 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:01.214057 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:01.317735 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:01.321883 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:01.326489 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:01.337129 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:01.347492 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:01.352066 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:01.360402 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:01.366688 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:01.370358 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:01.385353 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:01.476080 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:01.476188 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:01.476996 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:01.480501 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:01.485072 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:01.555716 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:07:01.556432 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:07:01.581405 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:01.584228 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:01.584667 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:01.588864 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:01.594794 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 08:07:01.595446 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 08:07:01.598935 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:01.663887 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:01.683271 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:01.683835 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:01.689121 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:01.694693 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:01.696857 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:01.798797 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240501 08:07:01.889250 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:01.889712 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:01.894546 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:01.899099 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:01.910822 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:01.987273 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:07:02.002751 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:02.003412 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:02.018564 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:02.025277 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:02.026049 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:02.032985 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:02.045826 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:02.050748 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:02.061576 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:02.064045 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:02.067706 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:02.081187 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:07:02.165482 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:02.169989 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:02.170836 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:02.174345 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:02.185582 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:02.274132 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:02.275267 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:02.275540 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:02.276556 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:02.283366 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:02.389837 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:02.397745 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:02.401206 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:02.404647 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:02.405740 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:02.420976 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:02.421451 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:02.423388 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:02.432905 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:02.457403 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:02.516065 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:02.749706 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:02.749825 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:02.752892 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:02.756429 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:02.762552 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:02.810443 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:02.818022 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:02.818130 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:02.819150 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:02.819411 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:02.916661 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:02.916975 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:02.917562 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:02.918313 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:02.918764 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:02.997326 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:03.028708 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:03.032534 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:03.037494 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:03.039743 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:03.040889 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:03.064025 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:03.122094 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:03.123431 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:03.124039 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:03.132709 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:03.142797 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:03.186954 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:03.186992 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:03.187260 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:03.194902 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:03.203799 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:03.297413 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:03.298962 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:03.299261 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:03.300654 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:03.303746 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:03.314502 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:03.347800 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:03.386893 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:03.412933 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:03.440174 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:07:03.441929 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:03.445101 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:03.446405 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:03.448809 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:03.448933 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:03.467824 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:03.486161 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:07:03.598034 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:03.601984 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:03.606307 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:03.608925 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:03.610594 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:03.622800 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:03.687297 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240501 08:07:03.709493 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:03.715693 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:03.718349 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:03.719417 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:03.720677 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:07:03.739990 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:03.784842 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:03.786396 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:03.791008 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:03.798120 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:03.799273 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:03.824785 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:07:03.875697 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:03.876785 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:03.879084 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:03.879678 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:03.880556 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:03.914686 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 08:07:03.965617 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 08:07:03.978386 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:03.978693 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:03.979655 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:03.986934 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:03.991535 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:04.071419 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:04.087281 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:04.093283 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:04.094167 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:04.096014 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:04.103595 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:04.155476 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:04.159144 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:04.161266 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:04.162230 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:04.163322 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:04.169852 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 08:07:04.387187 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:07:04.402060 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:04.402835 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:04.407541 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:04.411195 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:04.414602 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:04.421039 10768 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:07:04.421170 10768 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:07:04.421564 10768 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
W20240501 08:07:04.423507 13122 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:04.423720 13121 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:07:04.423777 13121 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9; no voters: 60e8857abe7c41038af46ebb5a095a90, ca177138f2c44d2eafa1646b795f7608
I20240501 08:07:04.424029 10768 raft_consensus.cc:2726] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:07:04.515380 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:04.515393 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:04.515679 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:04.518224 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:07:04.525467 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:04.539847 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:04.543790 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:04.544741 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:04.549151 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:04.556352 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:04.557272 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:04.572960 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:04.624060 4825 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:07:04.624197 4825 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:07:04.624663 4825 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
W20240501 08:07:04.625196 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:07:04.625928 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:04.626140 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:07:04.626190 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:07:04.626755 4825 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:07:04.662137 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:04.664191 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:04.665123 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:04.669062 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:04.674616 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:04.762720 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:04.763190 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:04.763482 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:04.764827 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:04.778970 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:04.805708 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:07:04.884177 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:04.890174 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:04.894089 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:04.896566 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:04.909502 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:04.910431 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:04.918109 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:04.919166 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:04.920529 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:04.937517 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:04.939627 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:05.239331 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:05.243655 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:05.244220 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:05.244225 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:05.267082 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:05.293143 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:05.299281 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:05.302613 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:05.303512 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:05.312803 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:05.382026 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:05.382601 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:05.384230 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:05.384479 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:05.392948 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:05.402696 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:05.486598 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:05.486613 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:05.488723 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:05.494598 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:05.498060 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:05.521152 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:05.570477 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:07:05.590354 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:05.590749 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:05.591070 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:05.596294 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:05.602846 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:05.654320 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:05.659379 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:05.659615 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:05.663205 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:05.663579 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:05.698897 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:05.729326 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:07:05.740458 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:05.741452 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:05.741757 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:05.741847 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:05.744529 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:05.808364 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:05.833015 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:05.853765 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 08:07:05.971001 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:05.971158 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:05.973589 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:07:05.976073 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:05.983137 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:05.986281 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:05.998555 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:06.033221 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:07:06.068243 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:06.071056 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:06.071945 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:06.074343 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:06.079228 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:07:06.080511 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:06.163082 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:06.163777 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:06.166626 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:06.167546 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:06.178067 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:06.266958 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240501 08:07:06.276275 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:06.281088 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:06.288311 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:06.292747 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:06.294333 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:06.314076 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:06.315269 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:06.322922 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:06.328771 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:06.339098 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:06.402825 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:06.403837 4825 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:07:06.403956 4825 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:07:06.404356 4825 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
W20240501 08:07:06.407223 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:06.407346 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:07:06.407393 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:07:06.407601 4825 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:07:06.419775 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:06.419829 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:06.422003 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:07:06.432791 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:06.434059 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:06.434324 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:06.451627 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 08:07:06.531749 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:06.533573 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:06.534507 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:06.534868 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:06.535059 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:06.634547 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:06.636305 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:06.644987 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:06.657310 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:06.658598 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:06.684053 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 08:07:06.684671 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 08:07:06.703440 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 08:07:06.803047 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240501 08:07:06.839916 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:06.839916 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:06.841820 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:06.843225 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:06.844327 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:06.903941 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:06.944988 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:06.956526 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:06.960034 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:06.970772 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:06.971079 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:06.985342 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:06.989282 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:07:06.993500 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:06.994993 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:07.006026 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:07.009443 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:07.089771 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:07.090394 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:07.092600 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:07.095145 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:07.097246 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:07.183848 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:07.186508 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:07.189097 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:07.191136 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:07.199009 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:07.286824 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:07.302448 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:07.304594 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:07.314778 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:07.315513 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:07.319475 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:07.356787 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:07.361374 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:07.364317 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:07.368687 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:07.372156 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:07.563993 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:07:07.631318 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:07.639993 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:07.642997 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:07.643491 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:07.649843 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:07.687402 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:07.700429 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:07.704186 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:07.711752 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:07.720506 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:07.721969 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:07.781790 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:07.784739 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:07.787252 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:07.788010 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:07.792304 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:07.892655 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:07.893464 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:07.893471 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:07.894068 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:07.896989 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:07.925212 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:07:08.001678 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:08.002079 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:08.005570 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:08.006022 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:08.013850 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:08.023382 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:07:08.062592 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:08.063323 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:08.063766 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:08.064610 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:08.072698 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:08.083549 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:07:08.144603 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:08.145308 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:08.147822 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:08.148775 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:08.155267 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:08.223078 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:08.260277 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:08.270891 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:08.307794 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 08:07:08.372068 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:08.373208 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:07:08.373486 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:08.373843 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:08.374636 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:08.387496 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:08.389489 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:08.395215 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:07:08.462236 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:08.463261 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:08.465768 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:08.474011 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:08.474822 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:08.557993 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:08.558163 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:08.563766 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:08.564407 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:08.565634 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:08.640043 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:08.673919 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:08.675062 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:08.677954 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:08.682283 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:08.689280 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:08.728533 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:08.730404 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:08.733098 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:08.736722 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:08.738680 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:08.758832 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:07:08.827824 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:08.829571 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:08.832723 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:08.833026 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:08.842613 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:08.863906 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 08:07:08.889503 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:07:08.932953 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:08.933434 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:08.935775 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:08.943622 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:08.947837 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:09.041366 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:09.043495 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:09.044955 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:09.050580 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:09.051486 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:09.113200 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240501 08:07:09.190953 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:09.240005 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:09.242538 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:09.243047 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:09.243415 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:09.254590 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:09.293078 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:09.312009 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 08:07:09.355059 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:09.368155 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:09.370000 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:09.370000 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:09.375422 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:09.389765 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:09.393086 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:09.393088 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:09.394982 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:09.402482 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:09.506338 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:09.507079 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:09.518617 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:09.520380 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:09.520979 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:09.610623 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:09.615383 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:09.618021 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:09.627743 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:07:09.628320 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 65 similar messages]
I20240501 08:07:09.628809 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:09.630779 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:09.718964 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:09.724619 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:09.725807 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:09.734360 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:09.754953 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:09.778628 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:09.784816 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:09.788872 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:09.790557 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:09.792585 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:09.794380 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:09.806371 11117 maintenance_manager.cc:419] P 2ed840e35a9f4afe9077d15510856d4e: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033357
I20240501 08:07:09.809984 11116 maintenance_manager.cc:643] P 2ed840e35a9f4afe9077d15510856d4e: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.003s user 0.001s sys 0.000s Metrics: {"bytes_written":2128,"delete_count":0,"lbm_write_time_us":136,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":73,"threads_started":1,"update_count":31,"wal-append.queue_time_us":176}
I20240501 08:07:09.810528 11117 maintenance_manager.cc:419] P 2ed840e35a9f4afe9077d15510856d4e: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.142771
I20240501 08:07:09.822650 11116 maintenance_manager.cc:643] P 2ed840e35a9f4afe9077d15510856d4e: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.012s user 0.008s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":15168,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":444,"dirs.run_cpu_time_us":102,"dirs.run_wall_time_us":129,"lbm_read_time_us":214,"lbm_reads_lt_1ms":17,"lbm_write_time_us":279,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":194,"threads_started":2,"update_count":31}
W20240501 08:07:09.890756 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:07:10.056339 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:10.059581 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:10.067677 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:10.078202 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:10.079063 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:10.131709 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:10.132948 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:10.136852 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:10.140713 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:10.141811 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:10.211556 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:10.237289 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:10.237601 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:10.237665 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:10.238292 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:10.238413 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:10.341023 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:10.342533 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:10.344523 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:10.355875 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:10.358659 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:10.374503 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:07:10.423110 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:10.428728 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:10.428941 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:10.436019 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:10.455955 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:10.462538 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:07:10.499121 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:10.499619 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:10.500798 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:10.513029 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:10.520155 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:10.580125 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:10.595528 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:10.598299 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:10.606582 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:10.606940 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:10.609869 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:10.609992 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:10.616863 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:10.641819 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:10.664163 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:10.728322 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:07:10.744663 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:10.747251 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:10.748435 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:10.748530 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:10.753376 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:10.886873 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 08:07:10.909219 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:10.912384 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:10.915066 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:10.915112 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:10.917017 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:11.017573 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:11.019017 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:11.019716 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:11.023896 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:11.029742 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:07:11.046504 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:11.086228 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:07:11.093621 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:11.095371 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:11.103217 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:11.122041 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:11.123117 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:11.160552 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:11.164125 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:11.164384 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:11.175756 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:11.175835 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:11.194224 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:11.260263 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:11.260717 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 08:07:11.263103 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:11.264098 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:11.264200 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:11.265794 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:11.365958 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:11.366550 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:11.366737 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:11.380594 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:11.381139 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:11.415776 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:11.421110 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20240501 08:07:11.454169 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:11.455686 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:11.456418 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:11.462643 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:11.464903 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:11.673748 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:11.674278 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:11.674371 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:11.683598 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:11.684432 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240501 08:07:11.686326 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:11.691495 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 08:07:11.692143 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:11.767570 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:11.770848 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:11.777109 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:11.777112 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:11.777962 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:11.797560 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:07:11.841167 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:11.842504 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:11.843446 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:11.844846 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:11.846391 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:11.855171 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 08:07:11.871630 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240501 08:07:11.945557 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:11.951328 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:11.955710 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:11.956264 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:11.958822 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:12.051512 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:12.054757 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:12.064702 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:12.065907 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:12.076205 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:12.114569 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:12.117545 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:12.123191 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:12.124588 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:12.135551 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:12.166687 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:12.232513 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:12.233393 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:12.234876 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:12.236661 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:12.243510 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:12.399616 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:07:12.440868 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:12.441658 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:12.445657 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:12.446048 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:12.448043 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:12.533253 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:12.534993 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:12.536363 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:12.537143 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:12.540009 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:12.612763 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:12.630194 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:12.633003 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:12.640559 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:12.640753 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:12.649052 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:12.741222 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:12.746094 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:12.746816 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:12.757323 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:12.791823 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:12.793448 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:12.796594 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:12.804299 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:12.806977 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:12.823772 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:12.870247 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:12.872155 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:07:12.900280 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:12.903167 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:12.904356 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:12.907475 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:12.915093 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:07:12.916998 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:13.010236 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:13.012346 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:13.012794 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:13.013768 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:13.023001 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:13.046355 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:07:13.110786 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:13.114130 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:13.114658 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:13.115906 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:13.116107 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:13.171277 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:13.213356 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:13.249222 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 08:07:13.271255 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:07:13.315909 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:13.316522 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:13.318404 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:13.326681 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:13.328958 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:13.405607 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:07:13.437000 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:13.440045 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:13.444438 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:13.448462 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:13.448892 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:13.463106 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:13.463332 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
I20240501 08:07:13.466581 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:13.467607 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:13.477236 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:13.477330 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:13.586761 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:13.586804 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:13.587842 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:13.589005 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:13.592201 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:07:13.598866 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:13.643326 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:13.686093 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:13.686726 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:13.692107 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:13.694829 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:13.698951 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:13.759621 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:07:13.798980 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:13.800709 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:13.805022 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:13.808759 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:13.811174 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:13.819877 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:13.829382 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:13.838820 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:13.839500 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:13.839900 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:14.086176 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:07:14.164956 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:14.172024 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:14.172827 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:14.182099 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:14.182767 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:14.194864 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:14.198195 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:14.207010 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:14.219909 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:14.221302 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:14.221293 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:14.304594 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:14.304703 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:14.310794 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:14.316295 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:14.318594 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:14.324445 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:07:14.413808 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:14.414706 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:14.414773 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:14.418686 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:14.420104 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:14.482277 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
I20240501 08:07:14.523770 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:14.525367 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:14.527089 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:14.528086 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:14.571218 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:14.574040 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:14.575618 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:14.581562 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:14.583789 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:14.588388 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 08:07:14.596105 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:14.645563 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:07:14.646250 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:14.686308 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:14.687422 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:14.687834 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:14.690204 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:14.690238 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:14.722276 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 08:07:14.869529 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:07:14.890661 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:14.890791 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:14.891414 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:14.895475 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:14.895927 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:14.993588 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:14.994594 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:14.995698 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:14.996873 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:14.997141 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:15.094784 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:15.098380 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:15.101646 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:15.102651 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:15.103282 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:15.208463 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:15.211721 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:15.219621 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:15.220204 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:15.227474 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:15.251259 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:15.251300 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:15.253141 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:15.257117 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:15.270192 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:07:15.271147 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:15.271354 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:15.338672 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:07:15.354749 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:15.359704 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:15.366994 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:15.373981 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:15.374935 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:15.464339 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:15.465014 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:07:15.466468 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:15.467432 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:15.471634 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:15.481285 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:15.499752 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:15.517913 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:07:15.561056 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:15.563253 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:15.564076 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:15.565030 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:15.571489 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:15.593501 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:15.656433 11114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 08:07:15.657079 11114 catalog_manager.cc:4573] TS 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c95c4f3ef08f44d1b69fb7b98426f5af: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
W20240501 08:07:15.662178 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:15.747792 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:07:15.772734 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:15.773891 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:15.776460 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:15.776942 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:15.780913 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:15.887006 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:15.893556 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:15.899554 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:15.900169 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:15.907785 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:15.908350 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:15.909976 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:15.915074 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:15.934553 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:15.949239 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:15.976315 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:15.988809 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
I20240501 08:07:16.037180 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:16.038926 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:16.039108 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:16.040892 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:16.041450 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:16.056954 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:07:16.132618 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:16.134995 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:16.135926 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:16.148193 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:16.150707 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:16.211534 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:16.259701 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:16.260659 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:16.265978 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:16.266994 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:16.276943 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:16.280184 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:16.280267 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:16.287931 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:16.303714 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:16.311852 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:16.327446 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:16.584079 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 08:07:16.618278 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:16.625281 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:16.629246 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:16.631431 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:16.635003 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:16.645982 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:16.646085 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:16.649453 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:16.649789 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:16.654660 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:16.666093 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:16.670807 11115 catalog_manager.cc:4573] TS e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ea97f34590684bcf998228245dbcd9b0: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
W20240501 08:07:16.694768 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 08:07:16.758294 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:16.758463 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:16.760181 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:16.761941 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:16.786999 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:16.870642 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:16.871256 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:16.873442 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:16.873720 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:16.874789 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:16.913353 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 08:07:16.913975 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 08:07:16.936863 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:16.965559 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20240501 08:07:16.978778 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:16.982776 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:16.983145 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:16.991601 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:17.001740 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:17.013722 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:17.014528 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:17.020665 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 08:07:17.023016 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:17.025669 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:17.026322 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:17.071022 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:17.116256 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:07:17.124183 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:17.124241 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:17.128666 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:17.129393 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:17.130900 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:17.323119 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:17.325750 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:17.326200 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:17.328083 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:17.328841 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:17.427786 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:17.428851 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:17.429453 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:17.429853 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:17.443470 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:17.522619 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:17.522718 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:17.525777 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:17.529824 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:17.532405 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:17.589656 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:17.656625 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:17.656617 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:17.657442 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:17.660800 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:17.662606 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:17.662976 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:17.664950 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:17.688513 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:17.697278 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:17.701959 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:07:17.711817 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:17.748973 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:07:17.788075 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:17.791241 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:17.791918 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:17.799677 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:17.799814 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:17.897646 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:17.898262 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:17.899458 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:17.899475 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:17.902277 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:17.972829 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:18.002068 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:18.004355 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:07:18.005834 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:18.006688 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:18.009114 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:18.014900 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:18.027941 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:18.065374 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:18.169143 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:18.170751 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:07:18.195295 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:18.197350 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:18.199339 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:18.207361 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:18.225976 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:18.319356 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:18.319859 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:18.326782 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:18.331241 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:18.333287 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:18.334237 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:18.342377 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:18.347698 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:07:18.363541 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:18.366403 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:18.381651 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:18.401314 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:18.452780 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:07:18.457003 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:18.457341 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:18.460942 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:18.461256 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:18.470656 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:18.541471 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:18.556123 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:18.556690 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:18.556835 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:18.559901 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:18.568892 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:18.652335 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
I20240501 08:07:18.684007 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:18.685931 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:18.686909 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:18.687851 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:18.695535 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:18.703874 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:18.703969 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:18.705734 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:18.720157 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:18.732146 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:18.978075 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:19.041173 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:19.042641 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:19.045974 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:19.055125 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:19.059226 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:19.064694 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:19.079270 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:19.083945 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:19.084064 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:19.107844 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:19.111894 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 08:07:19.182483 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:19.182544 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:19.185343 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:19.188483 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:19.195495 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:19.286890 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:19.287824 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:19.292001 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:19.299609 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:19.300539 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:19.322721 15171 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:07:19.322902 15171 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:07:19.323522 15171 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
W20240501 08:07:19.324651 13122 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:19.324887 13121 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:07:19.324957 13121 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9; no voters: 60e8857abe7c41038af46ebb5a095a90, ca177138f2c44d2eafa1646b795f7608
I20240501 08:07:19.325101 15171 raft_consensus.cc:2726] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 08:07:19.413204 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:19.415140 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:19.415232 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:19.415264 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:19.417919 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:19.433071 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:19.436839 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:19.440760 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:19.455324 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:19.471207 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:19.488283 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 08:07:19.506891 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:19.518071 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:19.520923 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:07:19.562162 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:19.562904 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:19.562913 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:19.564564 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:19.586918 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:19.595484 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 08:07:19.700656 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 08:07:19.751884 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:19.755681 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:19.756260 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:19.762357 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:19.764158 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:19.788542 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 08:07:19.870924 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:19.873564 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:19.874948 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:19.876468 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:19.885969 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:19.970101 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:19.973114 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:19.974644 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:19.975196 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:19.989362 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:20.014261 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:20.041559 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:20.081455 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:20.085140 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:20.088755 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:20.101533 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:20.101629 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:20.106613 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:20.118005 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:20.138805 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:20.138973 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:20.149552 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:20.228367 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:20.228798 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:20.231190 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:20.235134 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:20.236874 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:20.328343 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:20.328600 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:20.330651 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:20.330871 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:20.337054 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:20.405718 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:07:20.418728 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:20.418982 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:20.421604 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:20.434242 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:20.438657 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:20.464390 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:20.524081 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:20.553200 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:20.624984 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:07:20.629500 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:20.632745 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:20.633595 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:20.643067 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:20.646235 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:20.659632 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:20.667954 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:07:20.749775 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:20.754643 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:20.760121 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:20.766973 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:20.767407 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:20.777443 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:20.787837 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:20.787923 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:20.788321 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:20.807338 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:20.900696 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:20.905488 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:20.907205 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:20.909994 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:20.920328 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:20.920990 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:20.934579 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:07:21.004487 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:21.009514 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:21.011081 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:21.011834 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:21.015389 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:21.105288 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:07:21.107930 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:21.109874 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:21.112064 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:21.114676 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:21.122141 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:21.172755 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:21.173036 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:21.178596 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:21.186383 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:21.186383 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:21.245683 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:21.343256 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 08:07:21.359709 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:21.411695 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 08:07:21.416410 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:21.417176 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:21.426645 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:21.427772 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:21.447518 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:21.519515 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:21.522179 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:21.522974 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:21.523566 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:21.537858 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:21.617523 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:21.619299 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:21.619844 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:21.622038 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:21.622927 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:21.750555 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:21.752066 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:21.759258 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:21.760922 13166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:21.761135 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:21.763307 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:21.771927 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:21.779114 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:21.787565 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:21.810644 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:21.848836 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:21.885249 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:21.886277 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:21.888252 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:21.888876 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:21.898239 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:21.934288 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:21.944555 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 08:07:21.992408 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:21.994240 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:21.994266 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:21.996344 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:21.997924 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:22.030846 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240501 08:07:22.031744 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:22.037840 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 08:07:22.070111 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 08:07:22.103350 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:22.104420 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:22.105875 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:22.108681 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:22.114599 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:22.126866 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 08:07:22.258193 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:22.299435 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:22.299715 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:22.300042 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:22.309141 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:22.311298 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:22.405802 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:22.411389 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:22.414906 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:22.418226 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:22.434736 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:22.459682 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:22.463047 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:22.463595 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:22.472484 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:22.474965 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:22.566325 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:22.566947 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:22.567337 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:22.567533 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:22.574755 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:22.612530 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:22.668469 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:22.669755 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:22.671723 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:22.673063 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:22.673228 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:22.681470 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:22.753607 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:07:22.796183 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:22.796277 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:22.796424 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:22.804720 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:22.810496 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:22.822798 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:22.823004 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:22.850706 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:22.853369 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:22.853678 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:22.952595 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:23.062747 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:23.103127 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:07:23.156237 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:23.164589 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:23.166093 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:23.169514 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:23.174772 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:23.183064 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:23.187510 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:23.199244 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:23.201756 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:23.210592 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:23.239928 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:07:23.298626 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:23.300638 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:23.303483 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:23.304720 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:23.305128 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:23.311522 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:23.337064 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:23.338397 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 08:07:23.416769 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:23.417083 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:23.417754 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:23.418155 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:23.430603 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:23.506516 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:23.521652 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:23.524407 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:23.541385 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:23.545148 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:23.545333 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:23.548400 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:23.557018 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:23.557617 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:23.557920 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:23.591270 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:23.666231 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:07:23.676605 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:23.679250 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:23.679673 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:23.680367 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:23.682322 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:23.836081 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:07:23.866117 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:23.867547 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:23.868789 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:23.879271 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:23.892203 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:23.939951 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
I20240501 08:07:23.981221 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:23.982580 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:23.983568 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:23.983912 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:23.984362 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:24.017737 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:24.085075 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:24.085198 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:24.085470 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:24.086346 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:24.094825 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:24.095496 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:24.190523 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:24.195290 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:24.199126 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:24.205168 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:24.212018 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:24.213275 18459 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:07:24.213387 18459 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:07:24.213897 18459 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
W20240501 08:07:24.215371 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:24.215476 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:07:24.215507 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:07:24.215948 18459 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:07:24.230250 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:24.231175 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:24.246554 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:24.255048 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:24.257936 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:24.340584 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:24.342943 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:24.346218 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:07:24.349184 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:24.350926 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:24.358847 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:24.391113 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:07:24.458528 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:24.459642 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:24.461606 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:24.462571 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:24.468894 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:24.475167 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 08:07:24.522039 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:24.552984 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:24.553704 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:24.555546 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:24.567622 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:24.567606 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:24.624167 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 08:07:24.740857 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 08:07:24.775163 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:24.777362 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:24.778964 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:24.782819 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:24.783773 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:24.890091 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:24.896854 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:24.897503 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:24.902110 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:24.926173 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:24.937141 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:24.937286 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:24.937541 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:24.950167 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:24.951009 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:25.052927 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:25.058074 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:25.060144 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:25.061072 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:25.062682 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:25.067399 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:07:25.068770 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:25.159391 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:25.159547 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:25.162297 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:25.171085 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:25.173835 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:25.202284 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 08:07:25.274652 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:25.277841 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:25.282720 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:25.284375 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:25.291961 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:25.315987 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:25.318850 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:07:25.319280 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:25.319485 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:25.328666 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:25.338181 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:25.592448 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:25.610797 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:25.611203 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:25.621126 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:25.621704 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:25.625911 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:25.681479 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:25.682276 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:25.687402 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:25.689352 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:07:25.694983 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:25.698164 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:25.700181 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:07:25.784060 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:25.784051 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:25.784315 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:25.793018 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:25.795845 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:07:25.798581 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:25.897056 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:25.901238 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:25.903734 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:25.921761 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:25.925818 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:25.963169 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:25.968735 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:25.973137 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:25.975428 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:25.997406 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:26.010331 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:26.021378 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:07:26.047441 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:26.049010 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:26.053225 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:26.061941 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:26.071302 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 08:07:26.073714 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:26.083163 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:26.125761 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:07:26.162619 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:26.164032 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:26.166779 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:26.167352 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:26.179095 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:26.287245 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:26.287442 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:26.289208 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:26.294343 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:26.306816 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:26.328608 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 08:07:26.349548 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:26.377588 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:07:26.467103 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:26.471035 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:26.480422 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:26.480422 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:26.484145 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:26.583827 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:26.588472 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:26.591511 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:26.602181 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:26.611616 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:26.642772 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:26.659910 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:26.660792 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:26.662883 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:26.664181 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:26.669397 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:26.746177 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:26.746163 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:26.747026 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:26.749900 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:26.756125 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:26.858479 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:26.858819 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:26.860175 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:26.860399 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:26.861584 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:26.979423 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:26.986646 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:26.987126 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:26.993125 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:07:26.995649 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:27.000226 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:27.022789 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:27.023167 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:27.027905 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:27.043839 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 08:07:27.044622 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:27.052282 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 08:07:27.053815 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:07:27.055281 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:27.056394 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:27.109164 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 08:07:27.110597 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 08:07:27.193913 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:07:27.324963 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:27.327188 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:27.327365 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:27.329138 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:27.329923 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:27.401106 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:27.407404 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:27.412523 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:27.418493 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:27.419416 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:27.483903 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:27.484813 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:27.484993 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:27.486397 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:27.500082 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:27.515822 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:07:27.572589 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:27.592304 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:27.592674 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:27.593675 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:27.595889 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:27.614121 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:27.698974 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:27.704203 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:27.706616 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:27.721658 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:27.722378 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:27.773176 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:27.774811 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:27.777388 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:27.780248 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:27.797372 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:27.863392 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:27.863654 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:27.864004 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:27.866452 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:27.868711 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:27.940754 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:07:28.086339 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:28.089754 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:28.090061 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:28.090845 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:28.091953 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:28.105754 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:28.185514 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:28.185767 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:28.188802 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:28.189170 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:28.192333 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:28.284327 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:28.286772 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:28.287073 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:28.287747 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:28.296849 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:07:28.298696 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:28.385671 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:28.388386 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:28.394374 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:28.396513 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:28.409345 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:28.441427 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:07:28.448649 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:28.448771 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:28.457362 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:07:28.460238 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:28.468338 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:07:28.469645 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:28.473340 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:28.509624 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:07:28.530089 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:28.532750 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:28.538235 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:28.543011 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:28.543432 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:28.572561 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:28.630718 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:07:28.635465 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:28.637787 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:28.647989 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:28.648741 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:28.649003 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:28.767443 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:07:28.771562 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:28.773622 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:28.778071 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:28.785514 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:28.802722 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:28.877964 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240501 08:07:28.903504 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:07:28.946226 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:28.946506 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:28.950505 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:07:28.952081 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:28.962363 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:28.962507 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:29.048888 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:29.054160 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:29.054368 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:29.061590 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:29.067871 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:29.137007 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:29.140197 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:29.144362 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:29.146917 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:29.150579 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:29.168154 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:29.202870 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:29.204815 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:29.207971 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:29.214628 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:29.218336 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:29.304481 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:29.305562 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:29.305922 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:29.307099 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:29.309188 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:29.377899 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 08:07:29.402171 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:29.407754 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:29.415498 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:29.415604 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:29.418588 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:29.486876 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:29.487819 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:29.491684 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:29.501369 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:29.503041 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:29.515679 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:29.690449 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 08:07:29.695341 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:29.719084 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:29.720113 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:29.722913 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:29.727162 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:29.731184 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:29.748484 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:07:29.776074 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:07:29.837826 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:29.840155 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:29.848155 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:07:29.851869 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:29.857517 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:29.857853 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:29.898537 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:29.905368 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:29.907923 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:29.911195 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:29.912762 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:30.012203 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:30.014405 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:30.015448 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:30.022903 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:30.028604 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:30.113826 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:30.115167 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:30.125278 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:30.128453 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:30.130916 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:30.200354 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:30.200845 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:30.209273 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:30.209851 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:30.210489 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:30.233179 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:30.248736 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:07:30.252377 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:07:30.274626 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:30.274847 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:30.277612 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:30.279659 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:30.289705 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:30.527601 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:30.527901 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:30.530309 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:30.537338 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:30.540452 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:30.616088 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:30.616673 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:30.618710 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:30.620165 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:30.622095 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:30.683825 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:30.696727 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:07:30.719784 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:30.719889 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:30.721379 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:30.723861 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:30.736182 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:30.745551 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:30.746145 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:30.842779 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:30.842864 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:30.849192 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:30.856240 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:30.867265 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:30.886641 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:30.902084 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:30.902983 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:30.904090 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:30.910825 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:30.984616 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:30.986526 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:30.986922 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:30.990756 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:30.995407 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:31.031281 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:31.075610 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:07:31.094115 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:31.094848 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:31.095117 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:31.095629 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:31.096390 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:31.211452 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:31.212484 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:31.212736 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:31.222476 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:31.222781 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:31.241786 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:31.259606 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:31.282119 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 08:07:31.384531 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 08:07:31.405797 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:31.406653 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:31.408640 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:31.410298 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:31.410857 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:31.482180 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:31.518520 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:07:31.519382 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:31.520953 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:31.521425 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:31.521684 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:31.542600 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:31.574779 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:31.577463 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:31.578756 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:31.582718 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:31.591692 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:31.658828 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:31.673761 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:31.675981 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:31.679082 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:31.680547 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:31.695140 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:31.776150 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:31.779691 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:31.784910 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:31.794045 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:31.802636 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:31.904937 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:31.905113 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:31.914988 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:31.918239 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:31.921547 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:31.935612 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:31.938779 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:31.938879 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:31.953915 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:31.977850 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:31.991484 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 08:07:32.071480 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:07:32.088429 21402 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:07:32.088568 21402 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:07:32.089017 21402 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
W20240501 08:07:32.089978 13122 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:32.090294 13121 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:07:32.090366 13121 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9; no voters: 60e8857abe7c41038af46ebb5a095a90, ca177138f2c44d2eafa1646b795f7608
I20240501 08:07:32.090533 21402 raft_consensus.cc:2726] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:07:32.097622 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 08:07:32.098227 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 08:07:32.166105 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 08:07:32.191861 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:32.198819 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:32.273176 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:32.274827 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:32.288385 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:32.289543 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:32.314577 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:32.333775 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:32.337128 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:32.339406 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:32.350982 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:32.353241 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:32.441653 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:32.447223 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:32.447932 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:32.448752 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:32.454521 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:32.530196 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:07:32.556555 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:32.557219 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:32.560737 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:32.573392 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:32.574193 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:32.583295 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 08:07:32.614547 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:32.651149 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:32.653064 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:32.660315 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:32.666304 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:32.670872 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:32.718720 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:32.722581 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:32.724870 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:32.725314 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:32.725384 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:32.828691 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:32.829560 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:32.831288 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:32.841086 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:32.846810 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:32.961922 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:32.963552 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:32.965562 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:32.972049 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:32.974092 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:32.998030 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:33.025344 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:33.139911 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:33.142844 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:33.142863 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:33.144008 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:33.152343 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:07:33.155239 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:33.244015 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:33.244220 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:33.249395 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:33.255172 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:33.257542 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:33.309557 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:33.320735 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:33.324540 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:07:33.326743 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:33.327097 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:33.328146 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:33.336812 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:33.362782 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:33.396430 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:33.399178 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:33.401896 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:33.402525 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:33.403048 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:33.433914 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:33.497434 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:33.499183 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:33.499644 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:33.500303 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:33.510941 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:33.513676 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:33.612972 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:33.613736 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:33.615799 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:33.616531 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:33.625702 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:33.703083 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:33.704031 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:33.704916 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:33.707742 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:33.709632 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:33.789265 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:33.813413 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 08:07:33.895829 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:07:33.933946 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:33.937176 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:33.937176 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:33.949090 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:33.953723 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:34.054631 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:34.057960 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
I20240501 08:07:34.058773 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:34.066244 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:34.069315 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:34.080389 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:34.087350 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:34.093979 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:34.095899 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:34.102663 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:34.114063 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:34.119757 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:34.158654 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:07:34.207808 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:34.214612 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:34.216056 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:34.226487 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:34.231253 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:34.292515 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 08:07:34.316843 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:34.318620 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:34.321625 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:34.322261 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:34.330708 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:34.372123 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240501 08:07:34.392977 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:07:34.426613 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:34.433154 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:34.438632 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:34.438939 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:34.446170 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:34.478963 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:34.483968 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:34.498468 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:34.499970 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:34.500828 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:34.547518 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:34.607545 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:34.780481 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240501 08:07:34.806381 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:34.809075 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:34.810657 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:34.815373 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:34.826627 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:34.849188 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:34.852214 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:34.854629 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:34.858839 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:34.869330 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:34.964740 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:34.966382 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:34.966410 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:34.966811 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:34.975503 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:34.993099 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:35.075050 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:35.075798 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:35.076440 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:35.076694 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:35.076831 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:35.187005 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:35.189162 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:35.191650 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:35.195326 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:35.198009 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:35.199950 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:35.206210 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:35.215531 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:35.225178 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:35.229905 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:35.334774 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:35.337165 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:35.344565 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:35.346132 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:35.350523 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:35.353367 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:35.377481 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 08:07:35.423310 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:35.529884 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:35.532193 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:35.532685 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:35.538625 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:35.540235 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:35.597018 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:07:35.649400 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:35.650180 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:35.652534 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:35.660051 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:35.660835 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:35.746452 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:35.748603 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:35.748965 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:35.749244 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:35.751560 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:35.764958 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:35.807273 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:35.846877 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:07:35.855582 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:35.861979 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:35.869812 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:35.870734 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:07:35.872237 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:35.875186 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:35.911227 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:35.914507 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:35.916534 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:35.917183 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:35.925103 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:35.928081 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:35.939898 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:07:36.017858 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:36.018656 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:36.022385 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:36.027245 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:36.032491 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:36.126483 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:36.126994 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:36.135751 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:36.141947 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:36.144822 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:36.185003 21402 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:07:36.185139 21402 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:07:36.185596 21402 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
W20240501 08:07:36.186594 13122 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:36.186898 13121 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:07:36.186964 13121 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9; no voters: 60e8857abe7c41038af46ebb5a095a90, ca177138f2c44d2eafa1646b795f7608
I20240501 08:07:36.187098 21402 raft_consensus.cc:2726] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:07:36.210156 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:07:36.231073 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:36.234582 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:36.235828 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:36.238746 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:36.239400 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:36.354115 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:36.392766 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 08:07:36.420367 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:07:36.443089 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:36.444830 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:36.449888 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:36.455509 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:36.465106 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:36.557340 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:36.559705 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:36.569176 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:36.573354 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:36.581557 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:36.582950 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:36.586232 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:36.598268 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:36.612138 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:36.612318 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:36.644428 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:36.695817 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
I20240501 08:07:36.711136 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:36.712487 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:36.714191 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:36.714582 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:36.719744 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 08:07:36.723371 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:36.807307 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:36.812163 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:36.812822 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:36.819768 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:36.826840 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:36.836616 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 08:07:36.857338 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:36.927590 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:36.928933 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:36.929962 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:36.932888 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:36.940310 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:36.975322 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:36.975409 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:36.977023 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:36.984077 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:36.986435 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:37.011250 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:37.180402 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 08:07:37.266675 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 08:07:37.285486 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:37.287212 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:37.294219 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:37.295553 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:37.298658 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:37.337661 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:37.340322 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:37.340570 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:37.344679 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:37.353255 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:37.433925 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:37.446055 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:37.447712 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:37.448347 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:37.456250 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:37.472796 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:37.554229 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:37.555141 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:37.559614 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:37.562556 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:37.579916 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:37.583738 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:07:37.584635 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:37.594900 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:07:37.667644 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:37.668516 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:37.670671 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:37.671154 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:37.681700 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:37.717954 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:37.724413 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:37.724592 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:37.726074 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:37.729075 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:37.831519 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:37.838670 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:37.839588 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:37.839651 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:37.840138 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:37.952621 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:07:37.988266 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:37.989032 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:37.989121 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:37.991362 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:37.991636 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:38.007113 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:38.039196 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:38.136734 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:38.137696 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:38.137845 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:38.139137 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:38.143200 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:38.245453 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:38.246222 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:38.252406 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:38.253116 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:38.268316 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:38.337229 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:38.342125 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:38.342379 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:38.344823 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:38.363381 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:38.377507 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:38.397239 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:07:38.397370 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:38.400478 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:38.402844 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:38.404865 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:38.405510 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:38.494264 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:38.496640 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:38.498893 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:38.504184 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:38.510581 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:38.577647 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:38.580304 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:07:38.599337 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:38.601549 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:38.602586 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:38.602872 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:38.608314 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:38.623268 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:38.623754 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:07:38.686849 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:38.693269 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:38.693979 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:38.699121 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:38.700338 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:38.744237 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:07:38.918004 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:38.918730 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:38.919100 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:38.920610 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:38.922101 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:38.938664 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 08:07:39.029865 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:39.032696 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:39.035104 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:39.037448 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:39.041880 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:39.042471 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:39.052902 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:39.058496 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:39.064702 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:39.082140 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:39.158906 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:39.166216 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
I20240501 08:07:39.176795 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:39.181931 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:39.183650 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:39.184324 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:39.190616 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:39.263783 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 08:07:39.288743 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:07:39.294528 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:39.296456 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:39.298835 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:39.308296 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:39.319762 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:39.341856 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:39.363467 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
I20240501 08:07:39.381650 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:39.384070 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:39.387043 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:39.404937 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:39.416647 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:39.447073 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:39.448403 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:39.462061 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:39.463747 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:39.470638 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:39.698068 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:39.719141 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:39.724452 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:39.729709 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:39.738224 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:39.745186 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:39.779635 23890 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:07:39.779770 23890 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:07:39.780179 23890 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
W20240501 08:07:39.781390 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:07:39.782735 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:39.782853 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:07:39.782914 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:07:39.783041 23890 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:07:39.801313 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:39.803227 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:39.804630 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:39.806747 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:39.806977 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:39.904325 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:39.904835 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:39.908748 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:39.914790 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:39.920174 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:40.009083 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:07:40.024751 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:40.025131 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:40.027616 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:40.030475 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:40.030584 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:40.082463 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:40.082538 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:40.083541 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:40.085181 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:40.090325 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:07:40.109737 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:40.159425 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:40.159955 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:40.166354 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:40.167400 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:40.179423 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:40.199841 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:40.265826 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:40.265969 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:40.271351 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:40.274358 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:40.275230 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:40.393081 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:07:40.398291 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:40.398610 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:40.399878 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:40.401794 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:40.407389 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:40.441938 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:07:40.571233 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:40.571501 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:40.572361 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:40.574599 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:40.575922 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:40.594698 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 08:07:40.598099 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:07:40.682636 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:40.686589 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:40.693922 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:40.697161 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:40.697386 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:40.741133 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:40.747288 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:40.750718 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:40.751458 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:40.764839 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:40.769922 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:40.837762 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:07:40.846279 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:40.846760 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:40.847335 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:40.847987 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:07:40.849860 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:40.851356 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:40.887431 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:07:40.941840 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:40.949441 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:40.952899 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:40.955710 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:40.962759 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:41.057852 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:41.063021 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:41.076390 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:41.087231 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:41.088371 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:41.098945 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:41.099709 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:41.100064 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:41.100446 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:41.126364 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:41.150087 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:41.269310 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 08:07:41.307022 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:41.327733 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:07:41.409418 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:41.412273 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:41.412647 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:41.417651 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:41.422808 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:41.484701 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:41.487754 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:41.489209 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:41.491113 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:41.500391 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:41.560972 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:41.561806 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:41.562183 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:41.563861 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:41.578759 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:41.613224 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240501 08:07:41.641399 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
I20240501 08:07:41.667313 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:41.670102 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:41.670680 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:41.671084 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:41.695120 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:41.781253 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:41.784673 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:41.784694 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:41.793453 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:41.806988 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:41.808568 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:41.808645 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 08:07:41.844823 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:41.845669 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:41.853369 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:41.853369 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:41.855675 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:41.881747 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:07:41.947516 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:41.949465 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:41.950371 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:41.950971 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:41.951525 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:41.983836 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:07:42.150727 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:42.153700 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:42.154486 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:42.159622 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:42.162220 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:42.246158 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240501 08:07:42.252735 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:42.253109 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:42.253456 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:42.253712 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:42.255434 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:42.353166 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:42.353989 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:42.356408 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:42.358479 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:42.358867 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:42.375257 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 08:07:42.392789 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:42.424466 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:42.473521 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:07:42.481441 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:42.484617 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:42.486688 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:42.498030 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:42.506191 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:42.523622 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:42.527871 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:42.528317 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:42.542527 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:42.552517 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:42.636181 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:42.637640 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:42.644937 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:42.649084 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:42.664969 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:42.744969 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:42.745563 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:42.746773 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:42.749119 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:42.764941 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:42.769702 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:07:42.848623 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:42.849849 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:42.851073 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:42.851863 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:42.859688 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:42.932303 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:42.973645 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 08:07:43.009059 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 08:07:43.052930 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:07:43.057996 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:43.059698 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:43.063210 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:43.071111 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:43.071211 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:43.170190 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:43.170770 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:43.178653 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:43.186586 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:43.191895 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:43.195055 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:43.197494 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:43.197786 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:43.208945 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:43.210872 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:43.237848 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:07:43.318359 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:43.320565 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:43.320709 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:43.320812 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:43.322217 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:43.364599 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:43.378844 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:07:43.418041 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:43.419600 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:43.428200 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:43.428916 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:43.428978 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:07:43.432557 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:43.512465 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:07:43.535588 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:43.535881 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:43.552222 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:43.555485 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:43.556744 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:43.557690 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:43.557921 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:43.570410 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:43.596585 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:43.599653 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:43.744946 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:43.887467 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:43.896071 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:43.899693 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:43.906636 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:43.908741 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:43.914968 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:07:43.917232 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:43.918349 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:07:43.927188 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:43.947326 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:43.952291 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:43.954888 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:44.042739 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:44.044451 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:44.048128 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:44.051754 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:44.058703 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:44.134156 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
I20240501 08:07:44.152681 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:44.152738 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:44.153471 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:44.156311 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:44.159340 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:44.254159 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:44.257490 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:44.261444 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:44.265098 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:44.269392 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:44.275296 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 08:07:44.283885 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:44.297504 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:44.298151 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:44.303624 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:44.310606 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:44.325277 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:44.347379 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:44.383965 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
I20240501 08:07:44.415330 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:44.416817 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:44.417445 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:44.417825 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:44.421414 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:44.597025 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:44.597990 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:44.599287 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:44.604310 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:44.609562 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:07:44.611286 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:44.719964 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:44.722858 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:44.725126 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:44.726972 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:44.727033 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:44.789976 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 08:07:44.811760 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:44.818315 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:44.821084 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:44.823546 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:44.825266 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:44.848752 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:07:44.850289 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:44.921730 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:44.921762 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:44.925415 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:44.930950 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:44.939138 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:44.977416 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:44.980958 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:44.983711 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:44.987087 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:44.992851 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:07:44.993623 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:45.029508 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:07:45.078663 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:45.083608 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:45.086326 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:45.087975 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:45.097638 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:45.181165 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:45.183864 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:45.191927 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:45.192404 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:45.193981 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:45.278862 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:45.279023 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:45.280269 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:45.288646 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:45.289621 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:45.303701 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:45.328701 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:45.373128 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:07:45.490689 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:45.494074 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:45.495733 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:45.499148 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:45.505748 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:45.549712 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:07:45.603082 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:45.604573 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:45.607088 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:45.623270 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:45.624073 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:45.624976 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:45.634958 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:45.642927 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:45.644254 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:45.654597 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:45.718845 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:45.738788 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:07:45.760174 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:45.761561 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:45.766813 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:45.769821 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:45.771399 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:45.860540 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:45.862496 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:45.863076 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:45.867237 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:45.868418 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:45.886606 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:07:45.890149 23890 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:07:45.890265 23890 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:07:45.890677 23890 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
W20240501 08:07:45.891902 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:45.892012 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:07:45.892053 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:07:45.892406 23890 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:07:45.986447 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:45.987629 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:45.995662 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:45.998601 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:46.000509 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:46.001273 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:07:46.007083 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:46.016491 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:46.017040 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:46.020545 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 08:07:46.021826 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:46.029595 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:46.111837 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:46.317584 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:46.318280 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:46.342217 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:46.342710 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:46.345275 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:46.372046 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:46.373430 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:46.374989 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:46.378252 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:46.392755 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:46.465940 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:46.467343 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:46.470188 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:46.471511 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:46.473395 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:46.563333 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 08:07:46.569577 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:46.572904 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:46.573683 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:46.577232 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:46.594970 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:46.621408 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:07:46.682564 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:46.683672 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:46.685516 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:46.694070 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:46.700350 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 08:07:46.714006 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:07:46.715157 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:46.723146 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:46.743754 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:46.743861 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:46.749024 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:46.751369 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:46.760682 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:46.826248 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:46.829430 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:46.838804 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:46.848677 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:46.855795 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:46.894625 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 08:07:46.924209 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
I20240501 08:07:47.057914 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:47.058948 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:47.060796 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:47.062812 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:47.066517 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:07:47.070937 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:47.120322 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:47.150270 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:47.154081 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:47.154625 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:47.168948 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:47.179487 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:47.256744 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:47.259469 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:47.262773 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:47.270834 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:47.273488 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:47.318392 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:07:47.380800 11512 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:47.381088 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:47.383963 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:47.394258 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:47.400000 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:47.400633 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:47.402581 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:47.414870 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:47.415406 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:47.432104 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:47.471905 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:07:47.485055 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:07:47.522197 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:47.522370 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:47.522804 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:47.523540 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:47.533051 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:47.550711 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 08:07:47.554855 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:47.557047 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:07:47.624197 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:47.624832 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:47.625581 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:47.627122 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:47.638166 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:47.683410 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:07:47.721927 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:47.723306 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:47.724094 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:47.726902 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:47.731801 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:47.894650 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:07:47.941551 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:47.941851 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:47.942521 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:47.945273 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:47.945940 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:47.954061 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:48.047935 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:48.050904 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 08:07:48.052220 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:48.053123 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:48.065433 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:48.075871 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:48.086205 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:48.089814 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:48.098778 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:48.103755 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:48.121593 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:48.123736 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:48.142154 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:48.164861 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:07:48.204319 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:48.205226 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:48.205474 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:48.206038 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:48.206247 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:48.299787 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:48.301542 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:48.302013 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:48.305512 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:48.309692 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:48.378495 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:48.411188 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:48.413128 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:48.417361 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:48.433174 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:48.448567 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:48.454037 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:48.461947 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:48.466925 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:48.481724 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:48.489027 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:48.537310 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:07:48.744464 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:48.744607 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:48.748800 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:48.749460 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:48.754351 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:48.824612 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:48.826669 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:48.826889 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:48.829725 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:48.833731 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:48.874042 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 08:07:48.885845 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:48.923161 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:48.925498 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:48.927398 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:48.929208 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:48.935895 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:49.036828 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:49.038594 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:49.039232 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:49.042743 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:49.050731 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:49.079367 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 08:07:49.105367 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:49.109392 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:49.111678 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:49.118155 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:49.129343 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:49.175154 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:49.177527 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:49.186985 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:49.187108 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:49.196234 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:49.283437 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:49.283555 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:49.288076 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:49.292964 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:49.293814 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:49.307644 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:07:49.313076 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:07:49.316450 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240501 08:07:49.336004 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
I20240501 08:07:49.420150 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:49.423319 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:49.425451 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:49.426509 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:49.426625 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:49.485993 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:49.587394 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:49.589726 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:49.592880 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:49.600862 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:49.601127 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:49.611281 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:07:49.685942 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:49.687961 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:49.689565 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:49.691073 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:49.691404 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:49.720546 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:07:49.806759 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:49.809983 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:49.815006 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:49.815706 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:49.818089 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:49.827332 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240501 08:07:49.831883 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:49.843739 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:49.850936 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:49.851999 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:49.856014 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:49.942049 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:49.952888 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:49.955363 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:49.955782 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:49.956838 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:50.031060 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:50.049003 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:50.049645 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:50.051960 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:50.053697 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:50.066112 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:50.081856 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:07:50.158373 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:50.159168 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:50.163748 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:50.164862 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:50.171128 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:50.237277 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:07:50.248911 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:07:50.317550 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:07:50.355188 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:50.356101 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:50.360405 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:50.362699 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:50.368914 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:50.456171 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:50.457298 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:50.460027 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:50.476441 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:50.478682 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:50.511207 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:50.516929 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:50.517242 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:50.520769 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:50.526966 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:50.591198 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:07:50.603904 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:50.607805 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:50.607898 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:50.615705 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:50.616791 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:50.623817 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:07:50.653331 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:50.719940 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:50.721302 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:50.722524 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:50.723821 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:50.731025 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:50.833369 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:50.838877 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:50.841873 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:50.854466 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:50.862581 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:50.862953 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:50.865808 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:50.891474 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:50.893734 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:50.899129 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:50.957024 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:07:51.030625 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:07:51.114488 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:51.173297 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:51.184535 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:51.184644 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:51.185349 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:51.199296 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:51.234046 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:07:51.237366 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:51.244060 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:51.244715 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:51.248172 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:51.254830 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:51.338938 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:51.339090 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:51.340473 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:51.341163 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:51.343365 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:51.408023 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 08:07:51.449674 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:51.453815 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:51.458125 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:51.462919 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:51.471235 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:51.540988 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:51.541056 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:51.544026 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:51.545879 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:07:51.546659 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:51.553544 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:51.603664 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:51.607038 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:51.607327 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:51.607736 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:51.610836 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:51.687032 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:51.704270 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:51.705765 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:51.706661 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:51.708643 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:51.714754 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:51.763804 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 08:07:51.844926 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:07:51.858870 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:51.860042 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:51.861578 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:51.869756 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:51.874413 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:51.908511 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
I20240501 08:07:52.008740 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:52.010936 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:52.012585 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:52.021510 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:52.025513 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:52.030707 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 08:07:52.075307 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:07:52.109517 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:52.112681 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:52.113345 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:52.114756 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:52.115216 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:52.170610 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:07:52.194383 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:52.196730 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:52.198261 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:52.209947 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:52.217571 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:52.220414 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:52.263338 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:52.265338 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:52.265908 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:52.266827 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:52.286396 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:52.364562 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:52.364838 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:52.370710 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:52.376880 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:52.378079 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:52.482584 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:52.484310 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:52.486513 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:52.489637 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:52.491914 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:52.558095 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:52.559839 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:52.560920 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:52.570595 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:52.571877 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:52.631450 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:07:52.666605 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:07:52.678336 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 08:07:52.700975 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:52.809312 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:52.809849 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:52.810333 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:52.813413 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:52.814874 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:52.852540 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:07:52.887881 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:07:52.899082 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:52.904364 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:52.908519 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:52.908788 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:52.924386 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:52.937438 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:07:52.958411 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:52.958844 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:52.974288 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:52.976341 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:52.979835 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:53.067314 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:53.067346 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:53.067886 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:53.074529 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:53.094535 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:53.144773 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:53.146906 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:07:53.170867 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:53.173954 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:53.174330 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:53.184813 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:53.197470 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:53.198495 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 08:07:53.278930 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:07:53.280352 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:53.281030 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:53.284436 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:53.285297 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:53.299345 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:53.335054 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:53.336800 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:53.339336 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:53.343189 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:53.350888 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:53.360952 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:07:53.392526 29702 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:07:53.392696 29702 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:07:53.393203 29702 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
W20240501 08:07:53.394151 13122 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:53.394268 13121 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:07:53.394341 13121 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9; no voters: 60e8857abe7c41038af46ebb5a095a90, ca177138f2c44d2eafa1646b795f7608
I20240501 08:07:53.394498 29702 raft_consensus.cc:2726] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:07:53.540764 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:07:53.606531 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:53.613744 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:53.614936 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:53.624814 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:53.626577 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:53.686800 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:53.687436 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:53.691915 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:53.695569 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:53.696678 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 08:07:53.697849 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:53.725216 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:53.789732 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:53.790009 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:53.792464 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:53.792762 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:53.795430 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:53.864621 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:07:53.900714 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:53.905154 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:53.910190 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:53.916424 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:53.923586 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:53.976279 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:53.980580 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:53.987533 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:53.988670 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:54.004065 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:54.049816 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:54.051223 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:54.051560 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:54.051600 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:54.062991 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:54.115314 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 08:07:54.132359 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:54.151724 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:54.151943 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:54.154862 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:54.157407 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:54.164544 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:54.289882 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:54.290351 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:54.297835 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:54.299261 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:54.306174 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:54.392603 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
I20240501 08:07:54.459812 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:54.464491 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:54.470340 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:54.475075 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:54.479115 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:54.511096 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:07:54.534145 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:07:54.573339 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:54.573999 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:54.575264 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:54.576982 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:54.577881 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:54.617889 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
I20240501 08:07:54.636039 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:54.638666 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:54.646214 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:54.649986 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:54.654605 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:54.660259 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:54.712888 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:54.714270 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:54.714679 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:54.729053 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:54.738302 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:54.821337 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:54.823201 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:54.827064 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:54.834677 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:54.834941 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240501 08:07:54.837517 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:54.896209 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:07:54.940353 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:54.940768 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:54.941736 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:54.950476 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:54.967191 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:55.000679 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:55.002223 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:55.003473 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:55.005095 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:55.021127 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:55.034144 29702 raft_consensus.cc:482] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:07:55.034271 29702 raft_consensus.cc:504] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "53d1d2c0767b4f6195b69758a3dd6cf9" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36771 } } peers { permanent_uuid: "60e8857abe7c41038af46ebb5a095a90" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 42805 } }
I20240501 08:07:55.034675 29702 leader_election.cc:288] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805)
W20240501 08:07:55.035724 13122 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:55.035851 13121 leader_election.cc:334] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111)
I20240501 08:07:55.035903 13121 leader_election.cc:302] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53d1d2c0767b4f6195b69758a3dd6cf9; no voters: 60e8857abe7c41038af46ebb5a095a90, ca177138f2c44d2eafa1646b795f7608
I20240501 08:07:55.036098 29702 raft_consensus.cc:2726] T 53193305f868433c852755e8a8a81766 P 53d1d2c0767b4f6195b69758a3dd6cf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 08:07:55.085230 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:55.174242 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:07:55.273520 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:55.275563 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:55.276858 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:55.282498 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:55.290567 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:55.344489 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:55.353456 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:55.357882 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:55.358314 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:55.370514 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:07:55.375110 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:55.398479 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:07:55.424028 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:55.424538 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:55.427564 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:55.431529 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:55.436278 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:55.522976 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:55.523820 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:55.527369 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:55.528574 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:55.546253 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:55.574370 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:55.628438 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:55.628510 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:07:55.633869 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:55.634497 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:55.643172 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:55.655100 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:07:55.671445 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:55.712921 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:55.715435 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:55.716992 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:55.719203 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:55.731505 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:55.804709 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:55.806005 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:55.807050 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:55.808914 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:55.826994 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:55.912515 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:07:55.925733 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:07:56.027310 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:56.028482 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:56.036020 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:56.039595 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:56.045675 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:56.120684 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:56.124574 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:56.126470 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:56.127696 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:56.136947 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:56.142084 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:07:56.188398 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 08:07:56.194046 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:56.221201 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:56.223459 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:56.224617 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:56.235854 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:56.239013 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:07:56.248216 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:56.279213 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:07:56.341866 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:56.353195 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:56.355523 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:56.359560 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:56.360080 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:56.371730 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:56.381597 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:56.388422 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:56.394291 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:56.409468 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:56.481168 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:56.481719 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:56.481748 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:56.482913 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:56.491593 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:56.581074 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:56.582857 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:56.584846 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:56.591399 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:56.594491 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:56.637850 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:56.668345 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
I20240501 08:07:56.700320 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:56.701972 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:56.702391 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:56.703212 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:56.706015 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:56.839821 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 08:07:56.884868 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:56.885978 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:56.887538 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:56.888964 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:56.890543 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:56.905092 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:07:56.991354 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:56.992580 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:57.001556 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:57.003894 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:57.013695 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:57.025182 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:57.056116 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:57.058301 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:57.061054 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:57.061869 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:57.062323 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:57.150699 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:57.152812 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:57.154165 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:57.154439 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:57.158607 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:57.250039 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:57.251000 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:57.251147 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:57.253281 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:57.272634 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:57.371493 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:57.373845 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:57.378530 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:57.386269 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:57.393271 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:57.400583 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:07:57.415536 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:57.418434 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:57.423942 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:57.425472 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:07:57.429956 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:57.431351 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:57.554940 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:57.577804 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 08:07:57.698565 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:07:57.716935 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:57.717162 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:57.717168 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:57.720018 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:57.723423 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:57.739585 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 47 similar messages]
I20240501 08:07:57.774936 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:07:57.781294 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:07:57.783465 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:57.783845 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:57.794250 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:57.794246 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:57.878598 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:57.878924 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:57.879079 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:57.887475 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:57.889894 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:57.924628 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 08:07:57.996973 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:57.997689 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:57.998119 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:58.013360 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:58.022508 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:07:58.023553 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:07:58.026435 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:58.063875 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:58.068773 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:58.074990 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:58.081105 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:58.091537 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:58.103996 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:58.142172 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:58.142580 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:58.145165 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:58.146283 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:58.147694 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:58.247385 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:58.249814 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:58.253126 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:58.253110 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:58.258826 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:07:58.352306 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:07:58.378899 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:58.382376 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:58.384318 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:58.389325 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:58.389667 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:58.393611 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:07:58.416275 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 08:07:58.514325 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:07:58.551198 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:58.552963 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:58.553041 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:58.563357 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:58.565865 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:58.649593 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:58.656872 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:58.668984 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:58.670413 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:58.679338 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:58.727370 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:58.733422 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:58.736842 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:58.741940 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:58.744129 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:58.760116 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:07:58.761649 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:58.766321 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 14 similar messages]
I20240501 08:07:58.807973 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:58.809705 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:58.813684 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:58.814381 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:58.816342 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:58.856245 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:07:58.909065 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:58.916421 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:58.916679 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:58.924410 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:58.925143 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:59.025725 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:59.027447 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:59.030395 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:59.038713 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:59.045086 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:59.095651 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:59.100015 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:59.100061 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:59.100680 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:59.104745 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:59.214787 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240501 08:07:59.246564 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:59.332067 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:59.334254 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:59.334690 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:59.346658 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:59.348886 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:59.366008 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 08:07:59.367827 31455 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:07:59.367952 31455 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:07:59.368347 31455 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
W20240501 08:07:59.369990 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:07:59.370101 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:07:59.370151 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:07:59.370411 31455 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:07:59.433809 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:07:59.449534 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:59.456586 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:59.456961 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:59.470351 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:59.480741 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:59.493980 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:59.494655 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:59.500391 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:59.501564 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:59.507194 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:59.606523 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:59.609401 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:59.610556 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:59.621160 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:59.621155 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:07:59.680030 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:07:59.711441 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:59.713569 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:59.713850 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:59.721015 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:59.721590 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:07:59.731760 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:07:59.758881 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:07:59.812002 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:59.814846 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:59.816841 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:59.821703 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:07:59.834753 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:59.882493 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:07:59.882753 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:07:59.885432 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:07:59.885694 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:07:59.891551 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:07:59.955679 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 08:08:00.055035 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:08:00.139562 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:00.145529 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:00.147078 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:00.147161 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:00.147171 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:00.157475 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
I20240501 08:08:00.221390 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:00.221843 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:00.222755 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:00.232270 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:00.233853 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:00.255077 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:08:00.322731 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:00.324179 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:00.325297 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:00.328552 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:00.334645 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:00.430059 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:08:00.458966 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:00.459578 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:00.461968 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:00.466594 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:00.472524 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:00.475044 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:00.494051 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:08:00.495416 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:00.506764 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:00.519039 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:00.519167 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:00.519384 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:00.594784 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:00.596540 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:00.598587 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:00.605211 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:00.605466 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:00.708195 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:00.709332 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:00.710572 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:00.711685 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:00.712502 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:08:00.724748 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:00.772006 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:08:00.815686 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:00.815752 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:00.817344 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:00.818377 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:00.820784 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:00.849143 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:00.877485 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:08:00.935611 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:01.004364 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:08:01.012517 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:01.012861 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:01.015311 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:01.016206 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:01.027148 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:01.119616 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:01.120297 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:01.124812 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:08:01.131342 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:08:01.131744 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:01.138602 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:01.174623 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:01.176927 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:01.180953 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:01.187206 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:01.191460 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:01.261610 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 08:08:01.266696 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 08:08:01.278945 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:01.279311 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:01.283234 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:01.286557 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:01.303098 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:01.313127 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:08:01.384670 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:01.384742 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:01.386854 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:01.388947 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:01.398769 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:01.497933 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:01.499363 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:01.500895 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:01.519209 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:01.528371 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:01.528779 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:01.530886 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:01.537123 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:01.539149 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:01.554294 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:01.623020 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 08:08:01.731134 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:08:01.823174 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
I20240501 08:08:01.839206 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:01.839990 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:01.845021 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:01.863404 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:01.881666 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:01.903074 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:01.904531 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:01.907542 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:01.910943 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:01.921269 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:02.000839 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:08:02.005304 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:02.006997 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:02.009756 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:02.010380 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:02.021553 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:02.103232 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:02.108047 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:02.108479 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:02.108830 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:02.127302 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:02.148536 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:08:02.213500 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:02.214362 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:02.217274 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:02.227432 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:02.229517 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:08:02.240285 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:08:02.249708 11643 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:02.249970 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:02.253342 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:02.260690 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:02.266214 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:08:02.289798 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:08:02.364166 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:02.364917 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:02.367050 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:02.371004 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:02.376765 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:08:02.430910 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
I20240501 08:08:02.566830 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:02.567377 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:02.567867 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:02.570081 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:02.571007 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:02.632923 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:08:02.651221 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:08:02.674365 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:02.674953 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:02.675158 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:02.675794 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:02.676771 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:02.769973 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:02.778586 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:02.778985 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:02.779474 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:02.780869 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:02.837517 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 08:08:02.870699 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:08:02.887579 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:02.890628 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:02.897553 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:02.898176 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:02.906558 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:02.927512 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:02.932909 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:02.934963 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:02.944262 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:02.956239 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:03.039714 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:03.041956 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:03.044332 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:03.045738 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:03.058872 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:08:03.097245 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:03.113358 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:08:03.145068 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:03.145332 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:03.148659 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:03.149878 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:03.153563 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:03.162460 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 08:08:03.163193 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:08:03.246273 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:03.247071 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:03.250149 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:03.250866 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:03.250941 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:08:03.281795 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:03.433959 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:08:03.459230 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:03.459458 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:03.465190 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:03.477257 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:03.477440 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:03.557515 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:03.561309 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:03.567528 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:03.574772 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:03.586939 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:03.596957 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:08:03.605245 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:08:03.612794 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:03.612929 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:03.623121 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:03.630589 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:03.630637 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:03.639994 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:03.693749 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:08:03.726169 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:03.729650 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:03.730521 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:03.731163 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:03.731516 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:03.808288 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 08:08:03.835584 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:03.835673 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:03.848357 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:03.850554 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:03.854602 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:03.918237 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:03.924839 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:03.924932 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:03.925597 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:03.925819 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:03.999979 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:04.001922 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:04.002131 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:04.004588 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:04.023237 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:04.037395 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 08:08:04.091125 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:08:04.154589 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 08:08:04.235605 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:04.235845 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:04.236274 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:04.236730 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:04.237689 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:04.297173 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:08:04.325459 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:04.326683 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:04.327796 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:04.337270 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:04.344496 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:04.428129 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:04.429836 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:04.431385 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:04.432147 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:04.447154 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:04.528658 31455 raft_consensus.cc:482] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca177138f2c44d2eafa1646b795f7608)
I20240501 08:08:04.528811 31455 raft_consensus.cc:504] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "655ebda1fb2c49c4af7087a760529835" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41575 } } peers { permanent_uuid: "ca177138f2c44d2eafa1646b795f7608" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34503 } } peers { permanent_uuid: "e5c03261d31c4d9a91077a17e1cd0336" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 43995 } }
I20240501 08:08:04.529273 31455 leader_election.cc:288] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503), e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995)
W20240501 08:08:04.530643 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:08:04.530771 11337 leader_election.cc:334] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111)
I20240501 08:08:04.530825 11337 leader_election.cc:302] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 655ebda1fb2c49c4af7087a760529835; no voters: ca177138f2c44d2eafa1646b795f7608, e5c03261d31c4d9a91077a17e1cd0336
I20240501 08:08:04.530975 31455 raft_consensus.cc:2726] T f2e63d24d7df4fa3b2b26695e70ebe9f P 655ebda1fb2c49c4af7087a760529835 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:08:04.551239 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:04.552278 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:04.557924 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:04.563453 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:04.564994 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:04.566956 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:04.567124 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:08:04.568023 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:04.577230 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:08:04.596835 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
I20240501 08:08:04.609331 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:04.613864 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:04.678894 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:04.680297 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:04.680693 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:04.683269 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:04.684520 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:04.723461 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240501 08:08:04.764742 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:08:04.787873 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:04.792809 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:04.793275 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:04.794816 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:04.810988 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:04.904567 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:04.904707 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:08:04.907392 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:04.908161 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:04.908932 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:04.909170 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:05.027858 11336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 08:08:05.098642 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:05.099268 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:05.101434 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:05.102280 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:05.112396 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:05.133576 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:08:05.206888 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:05.207818 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:05.208767 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:05.210932 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:05.222798 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:08:05.229691 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:05.233834 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:08:05.263890 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:05.265933 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:05.266182 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:05.270872 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:05.275548 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:05.352420 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:05.354741 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:05.356148 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:05.359984 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:05.364660 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:05.452400 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:05.452533 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:05.457491 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:05.460695 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:05.461069 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:05.563871 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:08:05.564960 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:08:05.566934 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:05.568409 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:05.568593 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:05.569022 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:05.618667 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:05.619004 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:05.623508 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:05.626616 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:05.647029 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:05.668751 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:08:05.673830 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:08:05.891728 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:08:05.893024 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:08:05.898171 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:05.898614 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:05.905877 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:05.919703 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:05.937742 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:08:05.977523 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:05.978595 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:05.981742 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:05.982846 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:05.992115 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:06.044710 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:06.045246 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:06.046581 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:06.049225 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:06.059612 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:06.132764 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:08:06.146301 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:08:06.149456 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:08:06.150749 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:06.153113 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:06.155571 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:06.158311 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:06.174949 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 08:08:06.243294 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:08:06.249747 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:06.255450 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:06.255502 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:06.259749 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:06.264811 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:06.325781 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:06.326472 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:06.330261 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:06.335729 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:06.346717 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 08:08:06.360483 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:06.393709 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:06.395475 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:06.397634 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:06.412958 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:06.415432 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:06.508376 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:08:06.660355 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:06.666610 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:06.672426 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:08:06.676187 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 08:08:06.677009 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:06.680541 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:06.737301 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:06.738292 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:06.738538 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:06.739173 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:06.742587 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:06.794643 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:08:06.819341 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:08:06.832675 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:06.833395 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:06.845139 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:06.845597 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:06.849773 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:06.946808 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:06.955372 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:06.957336 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:06.958765 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:06.970494 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:07.005869 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
I20240501 08:08:07.015009 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:07.016055 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:07.027824 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:07.028720 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:07.049309 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:07.095783 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:07.097129 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:07.100700 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:07.101033 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:07.101961 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:08:07.192808 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:08:07.203789 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:07.204977 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:07.206372 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:07.207139 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:07.208245 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:07.335350 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
I20240501 08:08:07.335564 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:07.338737 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:07.339579 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:07.346877 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:07.353660 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:07.449389 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:08:07.507396 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:07.511011 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:07.511422 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:07.513303 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:07.515628 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:07.541678 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:07.587466 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:08:07.612675 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:07.613366 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:07.617784 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:07.629369 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:07.643748 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:07.696305 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:07.697139 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:07.701756 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:07.704759 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:07.723857 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:08:07.727401 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:08:07.774055 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:07.777333 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:07.785650 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:07.786729 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:07.787635 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:07.870214 11468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240501 08:08:07.880604 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:07.880623 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:07.883828 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:07.886608 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:07.897941 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:08:07.944818 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:08:07.993781 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:07.993907 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:08.004019 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:08.004736 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:08:08.005791 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:08:08.007241 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:08.020946 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:08:08.050670 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:08.052214 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:08.055807 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:08.057991 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:08.072130 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:08.225064 11206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 08:08:08.228988 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:08:08.323073 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:08.329186 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:08.329866 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:08.330933 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:08:08.339706 11468 consensus_peers.cc:485] T bd8348990f1b496bb62d8c0484193442 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:08:08.356364 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:08.409341 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:08.415457 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:08.417958 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:08:08.422632 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:08.423810 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:08.427238 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:08.482461 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:08.486186 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:08.486570 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:08:08.487397 11468 consensus_peers.cc:485] T a24be92aec7544089c7818f2802b9c85 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:08:08.495638 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:08.505348 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:08.589563 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:08.596048 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:08.596865 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:08.601074 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:08.611590 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:08.693361 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:08.697681 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:08.698004 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:08.710666 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:08.715577 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:08.784711 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:08.785671 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:08:08.787081 11206 consensus_peers.cc:485] T b9431cf5853c4148ac38855b1c083d4f P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:08:08.787833 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:08.788898 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:08.791769 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:08.821180 11206 consensus_peers.cc:485] T c95c4f3ef08f44d1b69fb7b98426f5af P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:08:08.848827 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 08:08:08.853724 11337 consensus_peers.cc:485] T 64613ea537c842f29d810e829278394e P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:08:08.853987 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:08.856942 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:08.861357 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:08.873080 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:08:08.960361 11468 consensus_peers.cc:485] T 5e8550ea95984d08937b9b3e2af12bec P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 08:08:08.990801 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
I20240501 08:08:09.102211 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:09.102277 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:09.102331 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:09.104214 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:09.105712 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:09.136494 11468 consensus_peers.cc:485] T 90f11b5693d7456d8a604e6c8726e69a P 22ec12774a4047a8b859743dc73f8ff6 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 08:08:09.187276 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:09.188726 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:09.188920 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:09.189788 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:09.196827 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:09.247260 11336 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer 60e8857abe7c41038af46ebb5a095a90 (127.0.99.133:42805): Couldn't send request to peer 60e8857abe7c41038af46ebb5a095a90. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:42805: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 08:08:09.249118 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 08:08:09.284986 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:09.286468 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:09.288452 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:09.289331 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:09.289461 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:09.294195 11599 consensus_peers.cc:485] T f80db2e37bbc4716ad3a5200c61ee97b P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:08:09.405827 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:09.407866 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:09.411507 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:09.427104 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:09.433221 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:09.473202 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:09.474093 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:09.479454 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:09.481570 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:09.483752 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:09.560987 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:09.562237 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:09.562251 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:09.564023 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:09.584985 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:09.605688 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:08:09.648540 11337 consensus_peers.cc:485] T 3f6e4ae97e884219ac7d0d94997d9cc9 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:08:09.668630 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:09.673485 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:09.674639 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:09.675926 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:09.683611 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:09.774063 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:09.775527 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:09.785513 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:09.792655 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:09.792693 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:09.879344 11468 consensus_peers.cc:485] T 6d214be86534458091f54a386d7d3334 P 22ec12774a4047a8b859743dc73f8ff6 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:09.917757 11337 consensus_peers.cc:485] T a1ebf29682024645978b3caa2d4988f3 P 655ebda1fb2c49c4af7087a760529835 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
I20240501 08:08:09.978790 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:09.981303 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:09.982303 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:09.984899 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:09.991613 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:10.030365 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:08:10.053141 11337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 08:08:10.054471 11337 consensus_peers.cc:485] T 5c47aaeb16c945ae84ece505aada2a7c P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:08:10.082084 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:10.088862 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:10.106086 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:10.108628 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:10.111248 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:10.124094 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:10.135653 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:10.137560 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:10.147493 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:10.154518 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:10.236711 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:10.242861 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:10.243583 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:10.249223 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:10.249739 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 08:08:10.258071 13122 consensus_peers.cc:485] T 25cee7e51c7c438aa721d71fb82dc71e P 53d1d2c0767b4f6195b69758a3dd6cf9 -> Peer e5c03261d31c4d9a91077a17e1cd0336 (127.0.99.135:43995): Couldn't send request to peer e5c03261d31c4d9a91077a17e1cd0336. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:43995: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:08:10.340691 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:10.343231 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:10.343353 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:10.343883 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:10.345175 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:10.456104 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:10.456511 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:10.459113 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:10.460314 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:10.466483 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:10.478593 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:10.482285 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:10.496263 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:10.496321 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:10.499205 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 08:08:10.569239 11206 consensus_peers.cc:462] T 7a266f266c3d4fd49fa8b325f1045d61 P e2dafc1866314669ad0de8fb7bf7fd05 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111)
W20240501 08:08:10.647428 11599 consensus_peers.cc:485] T 867ba08a15b44d4c9e6fa92b0d518d1d P 603919bb02364c04b6c530245b8b02f2 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 08:08:10.691080 11337 consensus_peers.cc:485] T 164fd265561e47128e7ff585b8cfab07 P 655ebda1fb2c49c4af7087a760529835 -> Peer ca177138f2c44d2eafa1646b795f7608 (127.0.99.136:34503): Couldn't send request to peer ca177138f2c44d2eafa1646b795f7608. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34503: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 08:08:10.819532 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:10.820358 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:10.821238 11510 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:08:10.826119 11248 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:08:10.834599 13165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:08:10.838689 11642 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:08:10.840881 11380 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:08:10.863960 11380 tablet_servi